ViewMTN: File doc/Programming-Guide/prog-guide.sgml in revision [0713a548..]

Below is the file 'doc/Programming-Guide/prog-guide.sgml' from this revision. You can also download the file.

<!doctype linuxdoc system> <article> <title>Squid Programmers Guide</title> <author>Squid Developers</author> <date>$Id: prog-guide.sgml,v 1.53 2005/05/17 16:56:02 hno Exp $</date> <abstract> Squid is a WWW Cache application developed by the National Laboratory for Applied Network Research and members of the Web Caching community. Squid is implemented as a single, non-blocking process based around a BSD select() loop. This document describes the operation of the Squid source code and is intended to be used by others who wish to customize or improve it. </abstract> <toc>  <sect>Introduction The Squid source code has evolved more from empirical observation and tinkering, rather than a solid design process. It carries a legacy of being ``touched'' by numerous individuals, each with somewhat different techniques and terminology. Squid is a single-process proxy server. Every request is handled by the main process, with the exception of FTP. However, Squid does not use a ``threads package'' such has Pthreads. While this might be easier to code, it suffers from portability and performance problems. Instead Squid maintains data structures and state information for each active request. The code is often difficult to follow because there are no explicit state variables for the active requests. Instead, thread execution progresses as a sequence of ``callback functions'' which get executed when I/O is ready to occur, or some other event has happened. As a callback function completes, it is responsible for registering the next callback function for subsequent I/O. Note there is only a pseudo-consistent naming scheme. In most cases functions are named like <tt/moduleFooBar()/. However, there are also some functions named like <tt/module_foo_bar()/. Note that the Squid source changes rapidly, and some parts of this document may become out-of-date. If you find any inconsistencies, please feel free to notify <url url="mailto:squid-dev@squid-cache.org" name="the Squid Developers">. <sect1>Conventions Function names and file names will be written in a courier font, such as <tt/store.c/ and <tt/storeRegister()/. Data structures and their members will be written in an italicized font, such as <em/StoreEntry/. <sect>Overview of Squid Components Squid consists of the following major components <sect1>Client Side Here new client connections are accepted, parsed, and processed. This is where we determine if the request is a cache HIT, REFRESH, MISS, etc. With HTTP/1.1 we may have multiple requests from a single TCP connection. Per-connection state information is held in a data structure called <em/ConnStateData/. Per-request state information is stored in the <em/clientHttpRequest/ structure. <sect1>Server Side These routines are responsible for forwarding cache misses to other servers, depending on the protocol. Cache misses may be forwarded to either origin servers, or other proxy caches. Note that all requests (FTP, Gopher) to other proxies are sent as HTTP requests. <tt/gopher.c/ is somewhat complex and gross because it must convert from the Gopher protocol to HTTP. Wais and Gopher don't receive much attention because they comprise a relatively insignificant portion of Internet traffic. <sect1>Storage Manager The Storage Manager is the glue between client and server sides. Every object saved in the cache is allocated a <em/StoreEntry/ structure. While the object is being accessed, it also has a <em/MemObject/ structure. Squid can quickly locate cached objects because it keeps (in memory) a hash table of all <em/StoreEntry/'s. The keys for the hash table are MD5 checksums of the objects URI. In addition there is also a storage policy such as LRU that keeps track of the objects and determines the removal order when space needs to be reclaimed. For the LRU policy this is implemented as a doubly linked list. For each object the <em/StoreEntry/ maps to a cache_dir and location via sdirn and sfilen. For the "ufs" store this file number (sfilen) is converted to a disk pathname by a simple modulo of L2 and L1, but other storage drivers may map sfilen in other ways. A cache swap file consists of two parts: the cache metadata, and the object data. Note the object data includes the full HTTP reply---headers and body. The HTTP reply headers are not the same as the cache metadata. Client-side requests register themselves with a <em/StoreEntry/ to be notified when new data arrives. Multiple clients may receive data via a single <em/StoreEntry/. For POST and PUT request, this process works in reverse. Server-side functions are notified when additional data is read from the client. <sect1>Request Forwarding <sect1>Peer Selection These functions are responsible for selecting one (or none) of the neighbor caches as the appropriate forwarding location. <sect1>Access Control These functions are responsible for allowing or denying a request, based on a number of different parameters. These parameters include the client's IP address, the hostname of the requested resource, the request method, etc. Some of the necessary information may not be immediately available, for example the origin server's IP address. In these cases, the ACL routines initiate lookups for the necessary information and continues the access control checks when the information is available. <sect1>Authentication Framework These functions are responsible for handling HTTP authentication. They follow a modular framework allow different authentication schemes to be added at will. For information on working with the authentication schemes See the chapter Authentication Framework. <sect1>Network Communication These are the routines for communicating over TCP and UDP network sockets. Here is where sockets are opened, closed, read, and written. In addition, note that the heart of Squid (<tt/comm_select()/ or <tt/comm_poll()/) exists here, even though it handles all file descriptors, not just network sockets. These routines do not support queuing multiple blocks of data for writing. Consequently, a callback occurs for every write request. <sect1>File/Disk I/O Routines for reading and writing disk files (and FIFOs). Reasons for separating network and disk I/O functions are partly historical, and partly because of different behaviors. For example, we don't worry about getting a ``No space left on device'' error for network sockets. The disk I/O routines support queuing of multiple blocks for writing. In some cases, it is possible to merge multiple blocks into a single write request. The write callback does not necessarily occur for every write request. <sect1>Neighbors Maintains the list of neighbor caches. Sends and receives ICP messages to neighbors. Decides which neighbors to query for a given request. File: <tt/neighbors.c/. <sect1>IP/FQDN Cache A cache of name-to-address and address-to-name lookups. These are hash tables keyed on the names and addresses. <tt/ipcache_nbgethostbyname()/ and <tt/fqdncache_nbgethostbyaddr()/ implement the non-blocking lookups. Files: <tt/ipcache.c/, <tt/fqdncache.c/. <sect1>Cache Manager This provides access to certain information needed by the cache administrator. A companion program, <em/cachemgr.cgi/ can be used to make this information available via a Web browser. Cache manager requests to Squid are made with a special URL of the form <verb> cache_object://hostname/operation </verb> The cache manager provides essentially ``read-only'' access to information. It does not provide a method for configuring Squid while it is running. <sect1>Network Measurement Database In a number of situation, Squid finds it useful to know the estimated network round-trip time (RTT) between itself and origin servers. A particularly useful is example is the peer selection algorithm. By making RTT measurements, a Squid cache will know if it, or one if its neighbors, is closest to a given origin server. The actual measurements are made with the <em/pinger/ program, described below. The measured values are stored in a database indexed under two keys. The primary index field is the /24 prefix of the origin server's IP address. Secondly, a hash table of fully-qualified host names have have data structures with links to the appropriate network entry. This allows Squid to quickly look up measurements when given either an IP address, or a host name. The /24 prefix aggregation is used to reduce the overall database size. File: <tt/net_db.c/. <sect1>Redirectors Squid has the ability to rewrite requests from clients. After checking the access controls, but before checking for cache hits, requested URLs may optionally be written to an external <em/redirector/ process. This program, which can be highly customized, may return a new URL to replace the original request. Common applications for this feature are extended access controls and local mirroring. File: <tt/redirect.c/. <sect1>Autonomous System Numbers Squid supports Autonomous System (AS) numbers as another access control element. The routines in <tt/asn.c/ query databases which map AS numbers into lists of CIDR prefixes. These results are stored in a radix tree which allows fast searching of the AS number for a given IP address. <sect1>Configuration File Parsing The primary configuration file specification is in the file <tt/cf.data.pre/. A simple utility program, <tt/cf_gen/, reads the <tt/cf.data.pre/ file and generates <tt/cf_parser.c/ and <tt/squid.conf/. <tt/cf_parser.c/ is included directly into <tt/cache_cf.c/ at compile time. <sect1>Callback Data Allocator Squid's extensive use of callback functions makes it very susceptible to memory access errors. Care must be taken so that the <tt/callback_data/ memory is still valid when the callback function is executed. The routines in <tt/cbdata.c/ provide a uniform method for managing callback data memory, canceling callbacks, and preventing erroneous memory accesses. <sect1>Debugging Squid includes extensive debugging statements to assist in tracking down bugs and strange behavior. Every debug statement is assigned a section and level. Usually, every debug statement in the same source file has the same section. Levels are chosen depending on how much output will be generated, or how useful the provided information will be. The <em/debug_options/ line in the configuration file determines which debug statements will be shown and which will not. The <em/debug_options/ line assigns a maximum level for every section. If a given debug statement has a level less than or equal to the configured level for that section, it will be shown. This description probably sounds more complicated than it really is. File: <em/debug.c/. Note that <tt/debug()/ itself is a macro. <sect1>Error Generation The routines in <tt/errorpage.c/ generate error messages from a template file and specific request parameters. This allows for customized error messages and multilingual support. <sect1>Event Queue The routines in <tt/event.c/ maintain a linked-list event queue for functions to be executed at a future time. The event queue is used for periodic functions such as performing cache replacement, cleaning swap directories, as well as one-time functions such as ICP query timeouts. <sect1>Filedescriptor Management Here we track the number of filedescriptors in use, and the number of bytes which has been read from or written to each file descriptor. <sect1>Hashtable Support These routines implement generic hash tables. A hash table is created with a function for hashing the key values, and a function for comparing the key values. <sect1>HTTP Anonymization These routines support anonymizing of HTTP requests leaving the cache. Either specific request headers will be removed (the ``standard'' mode), or only specific request headers will be allowed (the ``paranoid'' mode). <sect1>Internet Cache Protocol Here we implement the Internet Cache Protocol. This protocol is documented in the RFC 2186 and RFC 2187. The bulk of code is in the <tt/icp_v2.c/ file. The other, <tt/icp_v3.c/ is a single function for handling ICP queries from Netcache/Netapp caches; they use a different version number and a slightly different message format. <sect1>Ident Lookups These routines support RFC 931 ``Ident'' lookups. An ident server running on a host will report the user name associated with a connected TCP socket. Some sites use this facility for access control and logging purposes. <sect1>Memory Management These routines allocate and manage pools of memory for frequently-used data structures. When the <em/memory_pools/ configuration option is enabled, unused memory is not actually freed. Instead it is kept for future use. This may result in more efficient use of memory at the expense of a larger process size. <sect1>Multicast Support Currently, multicast is only used for ICP queries. The routines in this file implement joining a UDP socket to a multicast group (or groups), and setting the multicast TTL value on outgoing packets. <sect1>Persistent Server Connections These routines manage idle, persistent HTTP connections to origin servers and neighbor caches. Idle sockets are indexed in a hash table by their socket address (IP address and port number). Up to 10 idle sockets will be kept for each socket address, but only for 15 seconds. After 15 seconds, idle socket connections are closed. <sect1>Refresh Rules These routines decide wether a cached object is stale or fresh, based on the <em/refresh_pattern/ configuration options. If an object is fresh, it can be returned as a cache hit. If it is stale, then it must be revalidated with an If-Modified-Since request. <sect1>SNMP Support These routines implement SNMP for Squid. At the present time, we have made almost all of the cachemgr information available via SNMP. <sect1>URN Support We are experimenting with URN support in Squid version 1.2. Note, we're not talking full-blown generic URN's here. This is primarily targeted towards using URN's as an smart way of handling lists of mirror sites. For more details, please see <url url="http://squid.nlanr.net/Squid/urn-support.html" name="URN support in Squid">. <sect>External Programs <sect1>dnsserver Because the standard <tt/gethostbyname(3)/ library call blocks, Squid must use external processes to actually make these calls. Typically there will be ten <tt/dnsserver/ processes spawned from Squid. Communication occurs via TCP sockets bound to the loopback interface. The functions in <tt/dns.c/ are primarily concerned with starting and stopping the dnsservers. Reading and writing to and from the dnsservers occurs in the IP and FQDN cache modules. <sect1>pinger Although it would be possible for Squid to send and receive ICMP messages directly, we use an external process for two important reasons: <enum> <item>Because squid handles many filedescriptors simultaneously, we get much more accurate RTT measurements when ICMP is handled by a separate process. <item>Superuser privileges are required to send and receive ICMP. Rather than require Squid to be started as root, we prefer to have the smaller and simpler <em/pinger/ program installed with setuid permissions. </enum> <sect1>unlinkd The <tt/unlink(2)/ system call can cause a process to block for a significant amount of time. Therefore we do not want to make unlink() calls from Squid. Instead we pass them to this external process. <sect1>redirector A redirector process reads URLs on stdin and writes (possibly changed) URLs on stdout. It is implemented as an external process to maximize flexibility. <sect>Flow of a Typical Request <enum> <item> A client connection is accepted by the <em/client-side/. The HTTP request is parsed. <item> The access controls are checked. The client-side builds an ACL state data structure and registers a callback function for notification when access control checking is completed. <item> After the access controls have been verified, the client-side looks for the requested object in the cache. If is a cache hit, then the client-side registers its interest in the <em/StoreEntry/. Otherwise, Squid needs to forward the request, perhaps with an If-Modified-Since header. <item> The request-forwarding process begins with <tt/protoDispatch/. This function begins the peer selection procedure, which may involve sending ICP queries and receiving ICP replies. The peer selection procedure also involves checking configuration options such as <em/never_direct/ and <em/always_direct/. <item> When the ICP replies (if any) have been processed, we end up at <em/protoStart/. This function calls an appropriate protocol-specific function for forwarding the request. Here we will assume it is an HTTP request. <item> The HTTP module first opens a connection to the origin server or cache peer. If there is no idle persistent socket available, a new connection request is given to the Network Communication module with a callback function. The <tt/comm.c/ routines may try establishing a connection multiple times before giving up. <item> When a TCP connection has been established, HTTP builds a request buffer and submits it for writing on the socket. It then registers a read handler to receive and process the HTTP reply. <item> As the reply is initially received, the HTTP reply headers are parsed and placed into a reply data structure. As reply data is read, it is appended to the <em/StoreEntry/. Every time data is appended to the <em/StoreEntry/, the client-side is notified of the new data via a callback function. <item> As the client-side is notified of new data, it copies the data from the StoreEntry and submits it for writing on the client socket. <item> As data is appended to the <em/StoreEntry/, and the client(s) read it, the data may be submitted for writing to disk. <item> When the HTTP module finishes reading the reply from the upstream server, it marks the <em/StoreEntry/ as ``complete.'' The server socket is either closed or given to the persistent connection pool for future use. <item> When the client-side has written all of the object data, it unregisters itself from the <em/StoreEntry/. At the same time it either waits for another request from the client, or closes the client connection. </enum> <sect>Callback Functions <sect>The Main Loop: <tt/comm_select()/ At the core of Squid is the <tt/select(2)/ system call. Squid uses <tt/select()/ or <tt/poll(2)/ to process I/O on all open file descriptors. Hereafter we'll only use ``select'' to refer generically to either system call. The <tt/select()/ and <tt/poll()/ system calls work by waiting for I/O events on a set of file descriptors. Squid only checks for <em/read/ and <em/write/ events. Squid knows that it should check for reading or writing when there is a read or write handler registered for a given file descriptor. Handler functions are registered with the <tt/commSetSelect/ function. For example: <verb> commSetSelect(fd, COMM_SELECT_READ, clientReadRequest, conn, 0); </verb> In this example, <em/fd/ is a TCP socket to a client connection. When there is data to be read from the socket, then the select loop will execute <verb> clientReadRequest(fd, conn); </verb> The I/O handlers are reset every time they are called. In other words, a handler function must re-register itself with <tt/commSetSelect/ if it wants to continue reading or writing on a file descriptor. The I/O handler may be canceled before being called by providing NULL arguments, e.g.: <verb> commSetSelect(fd, COMM_SELECT_READ, NULL, NULL, 0); </verb> These I/O handlers (and others) and their associated callback data pointers are saved in the <em/fde/ data structure: <verb> struct _fde { ... PF *read_handler; void *read_data; PF *write_handler; void *write_data; close_handler *close_handler; DEFER *defer_check; void *defer_data; }; </verb> <em/read_handler/ and <em/write_handler/ are called when the file descriptor is ready for reading or writing, respectively. The <em/close_handler/ is called when the filedescriptor is closed. The <em/close_handler/ is actually a linked list of callback functions to be called. In some situations we want to defer reading from a filedescriptor, even though it has data for us to read. This may be the case when data arrives from the server-side faster than it can be written to the client-side. Before adding a filedescriptor to the ``read set'' for select, we call <em/defer_check/ (if it is non-NULL). If <em/defer_check/ returns 1, then we skip the filedescriptor for that time through the select loop. These handlers are stored in the <em/FD_ENTRY/ structure as defined in <tt/comm.h/. <tt/fd_table[]/ is the global array of <em/FD_ENTRY/ structures. The handler functions are of type <em/PF/, which is a typedef: <verb> typedef void (*PF) (int, void *); </verb> The close handler is really a linked list of handler functions. Each handler also has an associated pointer <tt/(void *data)/ to some kind of data structure. <tt/comm_select()/ is the function which issues the select() system call. It scans the entire <tt/fd_table[]/ array looking for handler functions. Each file descriptor with a read handler will be set in the <tt/fd_set/ read bitmask. Similarly, write handlers are scanned and bits set for the write bitmask. <tt/select()/ is then called, and the return read and write bitmasks are scanned for descriptors with pending I/O. For each ready descriptor, the handler is called. Note that the handler is cleared from the <em/FD_ENTRY/ before it is called. After each handler is called, <tt/comm_select_incoming()/ is called to process new HTTP and ICP requests. Typical read handlers are <tt/httpReadReply()/, <tt/diskHandleRead()/, <tt/icpHandleUdp()/, and <tt/ipcache_dnsHandleRead()/. Typical write handlers are <tt/commHandleWrite()/, <tt/diskHandleWrite()/, and <tt/icpUdpReply()/. The handler function is set with <tt/commSetSelect()/, with the exception of the close handlers, which are set with <tt/comm_add_close_handler()/. The close handlers are normally called from <tt/comm_close()/. The job of the close handlers is to deallocate data structures associated with the file descriptor. For this reason <tt/comm_close()/ must normally be the last function in a sequence to prevent accessing just-freed memory. The timeout and lifetime handlers are called for file descriptors which have been idle for too long. They are further discussed in a following chapter.  <sect>Processing Client Requests To be written...  <sect>Storage Manager <sect1>Introduction The Storage Manager is the glue between client and server sides. Every object saved in the cache is allocated a <em/StoreEntry/ structure. While the object is being accessed, it also has a <em/MemObject/ structure. Squid can quickly locate cached objects because it keeps (in memory) a hash table of all <em/StoreEntry/'s. The keys for the hash table are MD5 checksums of the objects URI. In addition there is also a storage policy such as LRU that keeps track of the objects and determines the removal order when space needs to be reclaimed. For the LRU policy this is implemented as a doubly linked list. For each object the <em/StoreEntry/ maps to a cache_dir and location via sdirn and sfilen. For the "ufs" store this file number (sfilen) is converted to a disk pathname by a simple modulo of L2 and L1, but other storage drivers may map sfilen in other ways. A cache swap file consists of two parts: the cache metadata, and the object data. Note the object data includes the full HTTP reply---headers and body. The HTTP reply headers are not the same as the cache metadata. Client-side requests register themselves with a <em/StoreEntry/ to be notified when new data arrives. Multiple clients may receive data via a single <em/StoreEntry/. For POST and PUT request, this process works in reverse. Server-side functions are notified when additional data is read from the client. <sect1>Object storage To be written... <sect1>Object retrieval To be written...  <sect>Storage Interface <sect1>Introduction Traditionally, Squid has always used the Unix filesystem (UFS) to store cache objects on disk. Over the years, the poor performance of UFS has become very obvious. In most cases, UFS limits Squid to about 30-50 requests per second. Our work indicates that the poor performance is mostly due to the synchronous nature of <tt/open()/ and <tt/unlink()/ system calls, and perhaps thrashing of inode/buffer caches. We want to try out our own, customized filesystems with Squid. In order to do that, we need a well-defined interface for the bits of Squid that access the permanent storage devices. We also require tighter control of the replacement policy by each storage module, rather than a single global replacement policy. <sect1>Build structure The storage types live in squid/src/fs/ . Each subdirectory corresponds to the name of the storage type. When a new storage type is implemented configure.in must be updated to autogenerate a Makefile in squid/src/fs/$type/ from a Makefile.in file. configure will take a list of storage types through the <em/--enable-store-io/ parameter. This parameter takes a list of space seperated storage types. For example, --enable-store-io="ufs coss" . Each storage type must create an archive file <tt/in squid/src/fs/$type/.a . This file is automatically linked into squid at compile time. Each storefs must export a function named <tt/storeFsSetup_$type()/. This function is called at runtime to initialise each storage type. The list of storage types is passed through <tt/store_modules.sh/ to generate the initialisation function <tt/storeFsSetup()/. This function lives in <tt/store_modules.c/. An example of the automatically generated file: <verb> /* automatically generated by ./store_modules.sh ufs coss * do not edit */ #include "squid.h" extern STSETUP storeFsSetup_ufs; extern STSETUP storeFsSetup_coss; void storeFsSetup(void) { storeFsAdd("ufs", storeFsSetup_ufs); storeFsAdd("coss", storeFsSetup_coss); } </verb> <sect1>Initialization of a storage type Each storage type initializes through the <tt/storeFsSetup_$type()/ function. The <tt/storeFsSetup_$type()/ function takes a single argument - a <tt/storefs_entry_t/ pointer. This pointer references the storefs_entry to initialise. A typical setup function is as follows: <verb> void storeFsSetup_ufs(storefs_entry_t *storefs) { assert(!ufs_initialised); storefs->parsefunc = storeUfsDirParse; storefs->reconfigurefunc = storeUfsDirReconfigure; storefs->donefunc = storeUfsDirDone; ufs_state_pool = memPoolCreate("UFS IO State data", sizeof(ufsstate_t)); ufs_initialised = 1; } </verb> There are five function pointers in the storefs_entry which require initializing. In this example, some protection is made against the setup function being called twice, and a memory pool is initialised for use inside the storage module. Each function will be covered below. <sect2>done <verb> typedef void STFSSHUTDOWN(void); </verb> This function is called whenever the storage system is to be shut down. It should take care of deallocating any resources currently allocated. <verb> typedef void STFSPARSE(SwapDir *SD, int index, char *path); typedef void STFSRECONFIGURE(SwapDir *SD, int index, char *path); </verb> These functions handle configuring and reconfiguring a storage directory. Additional arguments from the cache_dir configuration line can be retrieved through calls to strtok() and GetInteger(). <em/STFSPARSE/ has the task of initialising a new swapdir. It should parse the remaining arguments on the cache_dir line, initialise the relevant function pointers and data structures, and choose the replacement policy. <em/STFSRECONFIGURE/ deals with reconfiguring an active swapdir. It should parse the remaining arguments on the cache_dir line and change any active configuration parameters. The actual storage initialisation is done through the <em/STINIT/ function pointer in the SwapDir. <verb> struct _SwapDir { char *type; /* Pointer to the store dir type string */ int cur_size; /* Current swapsize in kb */ int low_size; /* ?? */ int max_size; /* Maximum swapsize in kb */ char *path; /* Path to store */ int index; /* This entry's index into the swapDir array */ int suggest; /* Suggestion for UFS style stores (??) */ size_t max_objsize; /* Maximum object size for this store */ union { /* Replacement policy-specific fields */ #ifdef HEAP_REPLACEMENT struct { heap *heap; } heap; #endif struct { dlink_list list; dlink_node *walker; } lru; } repl; int removals; int scanned; struct { unsigned int selected:1; /* Currently selected for write */ unsigned int read_only:1; /* This store is read only */ } flags; STINIT *init; /* Initialise the fs */ STNEWFS *newfs; /* Create a new fs */ STDUMP *dump; /* Dump fs config snippet */ STFREE *freefs; /* Free the fs data */ STDBLCHECK *dblcheck; /* Double check the obj integrity */ STSTATFS *statfs; /* Dump fs statistics */ STMAINTAINFS *maintainfs; /* Replacement maintainence */ STCHECKOBJ *checkob; /* Check if the fs will store an object, and get the FS load */ /* These two are notifications */ STREFOBJ *refobj; /* Reference this object */ STUNREFOBJ *unrefobj; /* Unreference this object */ STCALLBACK *callback; /* Handle pending callbacks */ STSYNC *sync; /* Sync the directory */ struct { STOBJCREATE *create; /* Create a new object */ STOBJOPEN *open; /* Open an existing object */ STOBJCLOSE *close; /* Close an open object */ STOBJREAD *read; /* Read from an open object */ STOBJWRITE *write; /* Write to a created object */ STOBJUNLINK *unlink; /* Remove the given object */ } obj; struct { STLOGOPEN *open; /* Open the log */ STLOGCLOSE *close; /* Close the log */ STLOGWRITE *write; /* Write to the log */ struct { STLOGCLEANOPEN *open; /* Open a clean log */ STLOGCLEANWRITE *write; /* Write to the log */ void *state; /* Current state */ } clean; } log; void *fsdata; /* FS-specific data */ }; </verb> <sect1>Operation of a storage module Squid understands the concept of multiple diverse storage directories. Each storage directory provides a caching object store, with object storage, retrieval, indexing and replacement. Each open object has associated with it a <em/storeIOState/ object. The <em/storeIOState/ object is used to record the state of the current object. Each <em/storeIOState/ can have a storage module specific data structure containing information private to the storage module. <verb> struct _storeIOState { sdirno swap_dirn; /* SwapDir index */ sfileno swap_filen; /* Unique file index number */ StoreEntry *e; /* Pointer to parent StoreEntry */ mode_t mode; /* Mode - O_RDONLY or O_WRONLY */ size_t st_size; /* Size of the object if known */ off_t offset; /* current _on-disk_ offset pointer */ STFNCB *file_callback; /* called on delayed sfileno assignments */ STIOCB *callback; /* IO Error handler callback */ void *callback_data; /* IO Error handler callback data */ struct { STRCB *callback; /* Read completion callback */ void *callback_data; /* Read complation callback data */ } read; struct { unsigned int closing:1; /* debugging aid */ } flags; void *fsstate; /* pointer to private fs state */ }; </verb> Each <em/SwapDir/ has the concept of a maximum object size. This is used as a basic hint to the storage layer in first choosing a suitable <em/SwapDir/. The checkobj function is then called for suitable candidate <em/SwapDirs/ to find out whether it wants to store a given <em/StoreEntry/. A <em/maxobjsize/ of -1 means 'any size'. The specific filesystem operations listed in the SwapDir object are covered below. <sect2>initfs <verb> typedef void STINIT(SwapDir *SD); </verb> Initialise the given <em/SwapDir/. Operations such as verifying and rebuilding the storage and creating any needed bitmaps are done here. <sect2>newfs <verb> typedef void STNEWFS(SwapDir *SD); </verb> Called for each configured <em/SwapDir/ to perform filesystem initialisation. This happens when '-z' is given to squid on the command line. <sect2>dumpfs <verb> typedef void STDUMP(StoreEntry *e, SwapDir *SD); </verb> Dump the FS specific configuration data of the current <em/SwapDir/ to the given <em/StoreEntry/. Used to grab a configuration file dump from th <em/cachemgr/ interface. Note: The printed options should start with a space character to separate them from the cache_dir path. <sect2>freefs <verb> typedef void STFREE(SwapDir *SD); </verb> Free the <em/SwapDir/ filesystem information. This routine should deallocate <em/SD->fsdata/. <sect2>doublecheckfs <verb> typedef int STDBLCHECK(SwapDir *SD, StoreEntry *e); </verb> Double-check the given object for validity. Called during rebuild if the '-S' flag is given to squid on the command line. Returns 1 if the object is indeed valid, and 0 if the object is found invalid. <sect2>statfs <verb> typedef void STSTATFS(SwapDir *SD, StoreEntry *e); </verb> Called to retrieve filesystem statistics, such as usage, load and errors. The information should be appended to the passed <em/StoreEntry/ e. <sect2>maintainfs <verb> typedef void STMAINTAINFS(SwapDir *SD); </verb> Called periodically to replace objects. The active replacement policy should be used to timeout unused objects in order to make room for new objects. <sect2>callback <verb> typedef void STCALLBACK(SwapDir *SD); </verb> This function is called inside the comm_select/comm_poll loop to handle any callbacks pending. <sect2>sync <verb> typedef void STSYNC(SwapDir *SD); </verb> This function is called whenever a sync to disk is required. This function should not return until all pending data has been flushed to disk. <sect2>parse/reconfigure <sect2>checkobj <verb> typedef int STCHECKOBJ(SwapDir *SD, const StoreEntry *e); </verb> Called by <tt/storeDirSelectSwapDir()/ to determine whether the <em/SwapDir/ will store the given <em/StoreEntry/ object. If the <em/SwapDir/ is not willing to store the given <em/StoreEntry/ -1 should be returned. Otherwise, a value between 0 and 1000 should be returned indicating the current IO load. A value of 1000 indicates the <em/SwapDir/ has an IO load of 100%. This is used by <tt/storeDirSelectSwapDir()/ to choose the <em/SwapDir/ with the lowest IO load. <sect2>referenceobj <verb> typedef void STREFOBJ(SwapDir *SD, StoreEntry *e); </verb> Called whenever an object is locked by <tt/storeLockObject()/. It is typically used to update the objects position in the replacement policy. <sect2>unreferenceobj <verb> typedef void STUNREFOBJ(SwapDir *SD, StoreEntry *e); </verb> Called whenever the object is unlocked by <tt/storeUnlockObject()/ and the lock count reaches 0. It is also typically used to update the objects position in the replacement policy. <sect2>createobj <verb> typedef storeIOState * STOBJCREATE(SwapDir *SD, StoreEntry *e, STFNCB *file_callback, STIOCB *io_callback, void *io_callback_data); </verb> Create an object in the <em/SwapDir/ *SD. <em/file_callback/ is called whenever the filesystem allocates or reallocates the <em/swap_filen/. Note - <em/STFNCB/ is called with a generic cbdata pointer, which points to the <em/StoreEntry/ e. The <em/StoreEntry/ should not be modified EXCEPT for the replacement policy fields. The IO callback should be called when an error occurs and when the object is closed. Once the IO callback is called, the <em/storeIOState/ becomes invalid. <em/STOBJCREATE/ returns a <em/storeIOState/ suitable for writing on sucess, or NULL if an error occurs. <sect2>openobj <verb> typedef storeIOState * STOBJOPEN(SwapDir *SD, StoreEntry *e, STFNCB *file_callback, STIOCB *io_callback, void *io_callback_data); </verb> Open the <em/StoreEntry/ in <em/SwapDir/ *SD for reading. Much the same is applicable from <em/STOBJCREATE/, the major difference being that the data passed to <em/file_callback/ is the relevant <em/store_client/ . <sect2>closeobj <verb> typedef void STOBJCLOSE(SwapDir *SD, storeIOState *sio); </verb> Close an opened object. The <em/STIOCB/ callback should be called at the end of this routine. <sect2>readobj <verb> typedef void STOBJREAD(SwapDir *SD, storeIOState *sio, char *buf, size_t size, off_t offset, STRCB *read_callback, void *read_callback_data); </verb> Read part of the object of into <em/buf/. It is safe to request a read when there are other pending reads or writes. <em/STRCB/ is called at completion. If a read operation fails, the filesystem layer notifies the calling module by calling the <em/STIOCB/ callback with an error status code. <sect2>writeobj <verb> typedef void STOBJWRITE(SwapDir *SD, storeIOState *sio, char *buf, size_t size, off_t offset, FREE *freefunc); </verb> Write the given block of data to the given store object. <em/buf/ is allocated by the caller. When the write is complete, the data is freed through <em/free_func/. If a write operation fails, the filesystem layer notifies the calling module by calling the <em/STIOCB/ callback with an error status code. <sect2>unlinkobj <verb> typedef void STOBJUNLINK(SwapDir *, StoreEntry *); </verb> Remove the <em/StoreEntry/ e from the <em/SwapDir/ SD and the replacement policy. <sect1>Store IO calls These routines are used inside the storage manager to create and retrieve objects from a storage directory. <sect2>storeCreate() <verb> storeIOState * storeCreate(StoreEntry *e, STIOCB *file_callback, STIOCB *close_callback, void * callback_data) </verb> <tt/storeCreate/ is called to store the given <em/StoreEntry/ in a storage directory. <tt/callback/ is a function that will be called either when an error is encountered, or when the object is closed (by calling <tt/storeClose()/). If the open request is successful, there is no callback. The calling module must assume the open request will succeed, and may begin reading or writing immediately. <tt/storeCreate()/ may return NULL if the requested object can not be created. In this case the <tt/callback/ function will not be called. <sect2>storeOpen() <verb> storeIOState * storeOpen(StoreEntry *e, STFNCB * file_callback, STIOCB * callback, void *callback_data) </verb> <tt/storeOpen/ is called to open the given <em/StoreEntry/ from the storage directory it resides on. <tt/callback/ is a function that will be called either when an error is encountered, or when the object is closed (by calling <tt/storeClose()/). If the open request is successful, there is no callback. The calling module must assume the open request will succeed, and may begin reading or writing immediately. <tt/storeOpen()/ may return NULL if the requested object can not be openeed. In this case the <tt/callback/ function will not be called. <sect2>storeRead() <verb> void storeRead(storeIOState *sio, char *buf, size_t size, off_t offset, STRCB *callback, void *callback_data) </verb> <tt/storeRead()/ is more complicated than the other functions because it requires its own callback function to notify the caller when the requested data has actually been read. <em/buf/ must be a valid memory buffer of at least <em/size/ bytes. <em/offset/ specifies the byte offset where the read should begin. Note that with the Swap Meta Headers prepended to each cache object, this offset does not equal the offset into the actual object data. The caller is responsible for allocating and freeing <em/buf/ . <sect2>storeWrite() <verb> void storeWrite(storeIOState *sio, char *buf, size_t size, off_t offset, FREE *free_func) </verb> <tt/storeWrite()/ submits a request to write a block of data to the disk store. The caller is responsible for allocating <em/buf/, but since there is no per-write callback, this memory must be freed by the lower filesystem implementation. Therefore, the caller must specify the <em/free_func/ to be used to deallocate the memory. If a write operation fails, the filesystem layer notifies the calling module by calling the <em/STIOCB/ callback with an error status code. <sect2>storeUnlink() <verb> void storeUnlink(StoreEntry *e) </verb> <tt/storeUnlink()/ removes the cached object from the disk store. There is no callback function, and the object does not need to be opened first. The filesystem layer will remove the object if it exists on the disk. <sect2>storeOffset() <verb> off_t storeOffset(storeIOState *sio) </verb> <tt/storeOffset()/ returns the current _ondisk_ offset. This is used to determine how much of an objects memory can be freed to make way for other in-transit and cached objects. You must make sure that the <em/storeIOState->offset/ refers to the ondisk offset, or undefined results will occur. For reads, this returns the current offset of successfully read data, not including queued reads. <sect1>Callbacks <sect2><em/STIOCB/ callback <verb> void stiocb(void *data, int errorflag, storeIOState *sio) </verb> The <em/stiocb/ function is passed as a parameter to <tt/storeOpen()/. The filesystem layer calls <em/stiocb/ either when an I/O error occurs, or when the disk object is closed. <em/errorflag/ is one of the following: <verb> #define DISK_OK (0) #define DISK_ERROR (-1) #define DISK_EOF (-2) #define DISK_NO_SPACE_LEFT (-6) </verb> Once the The <em/stiocb/ function has been called, the <em/sio/ structure should not be accessed further. <sect2><em/STRCB/ callback <verb> void strcb(void *data, const char *buf, size_t len) </verb> The <em/strcb/ function is passed as a parameter to <tt/storeRead()/. The filesystem layer calls <em/strcb/ after a block of data has been read from the disk and placed into <em/buf/. <em/len/ indicates how many bytes were placed into <em/buf/. The <em/strcb/ function is only called if the read operation is successful. If it fails, then the <em/STIOCB/ callback will be called instead. <sect1>State Logging These functions deal with state logging and related tasks for a squid storage system. These functions are used (called) in <tt/store_dir.c/. Each storage system must provide the functions described in this section, although it may be a no-op (null) function that does nothing. Each function is accessed through a function pointer stored in the <em/SwapDir/ structure: <verb> struct _SwapDir { ... STINIT *init; STNEWFS *newfs; struct { STLOGOPEN *open; STLOGCLOSE *close; STLOGWRITE *write; struct { STLOGCLEANOPEN *open; STLOGCLEANWRITE *write; void *state; } clean; } log; .... }; </verb> <sect2><tt/log.open()/ <verb> void STLOGOPEN(SwapDir *); </verb> The <tt/log.open/ function, of type <em/STLOGOPEN/, is used to open or initialize the state-holding log files (if any) for the storage system. For UFS this opens the <em/swap.state/ files. The <tt/log.open/ function may be called any number of times during Squid's execution. For example, the process of rotating, or writing clean logfiles closes the state log and then re-opens them. A <em/squid -k reconfigure/ does the same. <sect2><tt/log.close()/ <verb> void STLOGCLOSE(SwapDir *); </verb> The <tt/log.close/ function, of type <em/STLOGCLOSE/, is obviously the counterpart to <tt/log.open/. It must close the open state-holding log files (if any) for the storage system. <sect2><tt/log.write()/ <verb> void STLOGWRITE(const SwapDir *, const StoreEntry *, int op); </verb> The <tt/log.write/ function, of type <em/STLOGWRITE/, is used to write an entry to the state-holding log file. The <em/op/ argument is either <em/SWAP_LOG_ADD/ or <em/SWAP_LOG_DEL/. This feature may not be required by some storage systems and can be implemented as a null-function (no-op). <sect2><tt/log.clean.start()/ <verb> int STLOGCLEANSTART(SwapDir *); </verb> The <tt/log.clean.start/ function, of type <em/STLOGCLEANSTART/, is used for the process of writing "clean" state-holding log files. The clean-writing procedure is initiated by the <em/squid -k rotate/ command. This is a special case because we want to optimize the process as much as possible. This might be a no-op for some storage systems that don't have the same logging issues as UFS. The <em/log.clean.state/ pointer may be used to keep state information for the clean-writing process, but should not be accessed by upper layers. <sect2><tt/log.clean.nextentry()/ <verb> StoreEntry * STLOGCLEANNEXTENTRY(SwapDir *); </verb> Gets the next entry that is a candidate for the clean log. Returns NULL when there is no more objects to log <sect2><tt/log.clean.write()/ <verb> void STLOGCLEANWRITE(SwapDir *, const StoreEntry *); </verb> The <tt/log.clean.write/ function, of type <em/STLOGCLEANWRITE/, writes an entry to the clean log file (if any). <sect2><tt/log.clean.done()/ <verb> void STLOGCLEANDONE(SwapDir *); </verb> Indicates the end of the clean-writing process and signals the storage system to close the clean log, and rename or move them to become the official state-holding log ready to be opened. <sect1>Replacement policy implementation The replacement policy can be updated during STOBJREAD/STOBJWRITE/STOBJOPEN/ STOBJCLOSE as well as STREFOBJ and STUNREFOBJ. Care should be taken to only modify the relevant replacement policy entries in the StoreEntry. The responsibility of replacement policy maintainence has been moved into each SwapDir so that the storage code can have tight control of the replacement policy. Cyclic filesystems such as COSS require this tight coupling between the storage layer and the replacement policy. <sect1>Removal policy API The removal policy is responsible for determining in which order objects are deleted when Squid needs to reclaim space for new objects. Such a policy is used by a object storage for maintaining the stored objects and determining what to remove to reclaim space for new objects. (together they implements a replacement policy) <sect2>API It is implemented as a modular API where a storage directory or memory creates a policy of choice for maintaining it's objects, and modules registering to be used by this API. <sect3>createRemovalPolicy() <verb> RemovalPolicy policy = createRemovalPolicy(cons char *type, cons char *args) </verb> Creates a removal policy instance where object priority can be maintained The returned RemovalPolicy instance is cbdata registered <sect3>policy.Free() <verb> policy->Free(RemovalPolicy *policy) </verb> Destroys the policy instance and frees all related memory. <sect3>policy.Add() <verb> policy->Add(RemovalPolicy *policy, StoreEntry *, RemovalPolicyNode *node) </verb> Adds a StoreEntry to the policy instance. datap is a pointer to where policy specific data can be stored for the store entry, currently the size of one (void *) pointer. <sect3>policy.Remove() <verb> policy->Remove(RemovalPolicy *policy, StoreEntry *, RemovalPolicyNode *node) </verb> Removes a StoreEntry from the policy instance out of policy order. For example when an object is replaced by a newer one or is manually purged from the store. datap is a pointer to where policy specific data is stored for the store entry, currently the size of one (void *) pointer. <sect3>policy.Referenced() <verb> policy->Referenced(RemovalPolicy *policy, const StoreEntry *, RemovalPolicyNode *node) </verb> Tells the policy that a StoreEntry is going to be referenced. Called whenever a entry gets locked. node is a pointer to where policy specific data is stored for the store entry, currently the size of one (void *) pointer. <sect3>policy.Dereferenced() <verb> policy->Dereferenced(RemovalPolicy *policy, const StoreEntry *, RemovalPolicyNode *node) </verb> Tells the policy that a StoreEntry has been referenced. Called when an access to the entry has finished. node is a pointer to where policy specific data is stored for the store entry, currently the size of one (void *) pointer. <sect3>policy.WalkInit() <verb> RemovalPolicyWalker walker = policy->WalkInit(RemovalPolicy *policy) </verb> Initiates a walk of all objects in the policy instance. The objects is returned in an order suitable for using as reinsertion order when rebuilding the policy. The returned RemovalPolicyWalker instance is cbdata registered Note: The walk must be performed as an atomic operation with no other policy actions intervening, or the outcome will be undefined. <sect3>walker.Next() <verb> const StoreEntry *entry = walker->Next(RemovalPolicyWalker *walker) </verb> Gets the next object in the walk chain Return NULL when there is no further objects <sect3>walker.Done() <verb> walker->Done(RemovalPolicyWalker *walker) </verb> Finishes a walk of the maintained objects, destroys walker. <sect3>policy.PurgeInit() <verb> RemovalPurgeWalker purgewalker = policy->PurgeInit(RemovalPolicy *policy, int max_scan) </verb> Initiates a search for removal candidates. Search depth is indicated by max_scan. The returned RemovalPurgeWalker instance is cbdata registered Note: The walk must be performed as an atomic operation with no other policy actions intervening, or the outcome will be undefined. <sect3>purgewalker.Next() <verb> StoreEntry *entry = purgewalker->Next(RemovalPurgeWalker *purgewalker) </verb> Gets the next object to purge. The purgewalker will remove each returned object from the policy. It is the polices responsibility to verify that the object isn't locked or otherwise prevented from being removed. What this means is that the policy must not return objects where storeEntryLocked() is true. Return NULL when there is no further purgeable objects in the policy. <sect3>purgewalker.Done() <verb> purgewalker->Done(RemovalPurgeWalker *purgewalker) </verb> Finishes a walk of the maintained objects, destroys walker and restores the policy to it's normal state. <sect3>policy.Stats() <verb> purgewalker->Stats(RemovalPurgeWalker *purgewalker, StoreEntry *entry) </verb> Appends statistics about the policy to the given entry. <sect2>Source layout Policy implementations resides in src/repl/<name>/, and a make in such a directory must result in a object archive src/repl/<name>.a containing all the objects implementing the policy. <sect2>Internal structures <sect3>RemovalPolicy <verb> typedef struct _RemovalPolicy RemovalPolicy; struct _RemovalPolicy { char *_type; void *_data; void (*add)(RemovalPolicy *policy, StoreEntry *); ... /* see the API definition above */ }; </verb> The _type member is mainly for debugging and diagnostics purposes, and should be a pointer to the name of the policy (same name as used for creation) The _data member is for storing policy specific information. <sect3>RemovalPolicyWalker <verb> typedef struct _RemovalPolicyWalker RemovalPolicyWalker; struct _RemovalPolicyWalker { RemovalPolicy *_policy; void *_data; StoreEntry *(*next)(RemovalPolicyWalker *); ... /* see the API definition above */ }; </verb> <sect3>RemovalPolicyNode <verb> typedef struct _RemovalPolicyNode RemovalPolicyNode; struct _RemovalPolicyNode { void *data; }; </verb> Stores policy specific information about a entry. Currently there is only space for a single pointer, but plans are to maybe later provide more space here to allow simple policies to store all their data "inline" to preserve some memory. <sect2>Policy registration Policies are automatically registered in the Squid binary from the policy selection made by the user building Squid. In the future this might get extended to support loadable modules. All registered policies are available to object stores which wishes to use them. <sect2>Policy instance creation Each policy must implement a "create/new" function "<tt/RemovalPolicy * createRemovalPolicy_<name>(char *arguments)/". This function creates the policy instance and populates it with at least the API methods supported. Currently all API calls are mandatory, but the policy implementation must make sure to NULL fill the structure prior to populating it in order to assure future API compability. It should also populate the _data member with a pointer to policy specific data. <sect2>Walker When a walker is created the policy populates it with at least the API methods supported. Currently all API calls are mandatory, but the policy implementation must make sure to NULL fill the structure prior to populating it in order to assure future API compatibility. <sect2>Design notes/bugs The RemovalPolicyNode design is incomplete/insufficient. The intention was to abstract the location of the index pointers from the policy implementation to allow the policy to work on both on-disk and memory caches, but unfortunately the purge method for HEAP based policies needs to update this, and it is also preferable if the purge method in general knows how to clear the information. I think the agreement was that the current design of tightly coupling the two together on one StoreEntry is not the best design possible. It is debated if the design in having the policy index control the clean index writes is the correct approach. Perhaps not. Perhaps a more appropriate design is probably to do the store indexing completely outside the policy implementation (i.e. using the hash index), and only ask the policy to dump it's state somehow. The Referenced/Dereferenced() calls is today mapped to lock/unlock which is an approximation of when they are intended to be called. However, the real intention is to have Referenced() called whenever an object is referenced, and Dereferenced() only called when the object has actually been used for anything good.  <sect>Forwarding Selection To be written...  <sect>IP Cache and FQDN Cache <sect1> Introduction The IP cache is a built-in component of squid providing Hostname to IP-Number translation functionality and managing the involved data-structures. Efficiency concerns require mechanisms that allow non-blocking access to these mappings. The IP cache usually doesn't block on a request except for special cases where this is desired (see below). <sect1> Data Structures The data structure used for storing name-address mappings is a small hashtable (static hash_table *ip_table), where structures of type ipcache_entry whose most interesting members are: <verb> struct _ipcache_entry { char *name; time_t lastref; ipcache_addrs addrs; struct _ip_pending *pending_head; char *error_message; unsigned char locks; ipcache_status_t status:3; } </verb> <sect1> External overview Main functionality is provided through calls to: <descrip> <tag>ipcache_nbgethostbyname(const char *name, IPH *handler, void *handlerdata)</tag> where <em/name/ is the name of the host to resolve, <em/handler/ is a pointer to the function to be called when the reply from the IP cache (or the DNS if the IP cache misses) and <em/handlerdata/ is information that is passed to the handler and does not affect the IP cache. <tag>ipcache_gethostbyname(const char *name,int flags)</tag> is different in that it only checks if an entry exists in it's data-structures and does not by default contact the DNS, unless this is requested, by setting the <em/flags/ to <em/IP_BLOCKING_LOOKUP/ or <em/IP_LOOKUP_IF_MISS/. <tag>ipcache_init()</tag> is called from <em/mainInitialize()/ after disk initialization and prior to the reverse fqdn cache initialization <tag>ipcache_restart()</tag> is called to clear the IP cache's data structures, cancel all pending requests. Currently, it is only called from <em/mainReconfigure/. </descrip> <sect1> Internal Operation Internally, the execution flow is as follows: On a miss, <em/ipcache_getnbhostbyname/ checks whether a request for this name is already pending, and if positive, it creates a new entry using <em/ipcacheAddNew/ with the <em/IP_PENDING/ flag set . Then it calls <em/ipcacheAddPending/ to add a request to the queue together with data and handler. Else, <em/ipcache_dnsDispatch()/ is called to directly create a DNS query or to <em/ipcacheEnqueue()/ if all no DNS port is free. <em/ipcache_call_pending()/ is called regularly to walk down the pending list and call handlers. LRU clean-up is performed through <em/ipcache_purgelru()/ according to the <em/ipcache_high/ threshold.  <sect>Server Protocols <sect1>HTTP To be written... <sect1>FTP To be written... <sect1>Gopher To be written... <sect1>Wais To be written... <sect1>SSL To be written... <sect1>Passthrough To be written...  <sect>Timeouts To be written...  <sect>Events To be written...  <sect>Access Controls To be written...  <sect>Authentication Framework Squid's authentication system is responsible for reading authentication credentials from HTTP requests and deciding whether or not those credentials are valid. This functionality resides in two separate components: Authentication Schemes and Authentication Modules. An Authentication Scheme describes how Squid gets the credentials (i.e. username, password) from user requests. Squid currently supports two authentication schemes: Basic and NTLM. Basic authentication uses the <em/WWW-Authenticate/ HTTP header. The Authentication Scheme code is implemented inside Squid itself. An Authentication Module takes the credentials received from a client's request and tells Squid if they are are valid. Authentication Modules are implemented externally from Squid, as child helper processes. Authentication Modules interface with various types authentication databases, such as LDAP, PAM, NCSA-style password files, and more. <sect1>Authentication Scheme API <sect2>Definition of an Authentication Scheme An auth scheme in squid is the collection of functions required to manage the authentication process for a given HTTP authentication scheme. Existing auth schemes in squid are Basic and NTLM. Other HTTP schemes (see for example RFC 2617) have been published and could be implemented in squid. The term auth scheme and auth module are interchangeable. An auth module is not to be confused with an authentication helper, which is a scheme specific external program used by a specific scheme to perform data manipulation external to squid. Typically this involves comparing the browser submitted credentials with those in the organization's user directory. Auth modules SHOULD NOT perform access control functions. Squid has advanced caching access control functionality already. Future work in squid will allow a auth scheme helper to return group information for a user, to allow Squid to more seamlessly implement access control. <sect2>Function typedefs Each function related to the general case of HTTP authentication has a matching typedef. There are some additional function types used to register/initialize, deregister/shutdown and provide stats on auth modules: <descrip> <tag/typedef int AUTHSACTIVE();/ The Active function is used by squid to determine whether the auth module has successfully initialised itself with the current configuration. <tag/typedef int AUTHSCONFIGURED();/ The configured function is used to see if the auth module has been given valid parameters and is able to handle authentication requests if initialised. If configured returns 0 no other module functions except Shutdown/Dump/Parse/FreeConfig will be called by Squid. <tag/typedef void AUTHSSETUP(authscheme_entry_t *);/ functions of type AUTHSSETUP are used to register an auth module with squid. The registration function MUST be named "authSchemeSetup_SCHEME" where SCHEME is the auth_scheme as defined by RFC 2617. Only one auth scheme registered in squid can provide functionality for a given auth_scheme. (I.e. only one auth module can handle Basic, only one can handle Digest and so forth). The Setup function is responsible for registering the functions in the auth module into the passed authscheme_entry_t. The authscheme_entry_t will never be NULL. If it is NULL the auth module should log an error and do nothing. The other functions can have any desired name that does not collide with any statically linked function name within Squid. It is recommended to use names of the form "authe_SCHEME_FUNCTIONNAME" (for example authenticate_NTLM_Active is the Active() function for the NTLM auth module. <tag/typedef void AUTHSSHUTDOWN(void);/ Functions of type AUTHSSHUTDOWN are responsible for freeing any resources used by the auth modules. The shutdown function will be called before squid reconfigures, and before squid shuts down. <tag/typedef void AUTHSINIT(authScheme *);/ Functions of type AUTHSINIT are responsible for allocating any needed resources for the authentication module. AUTHSINIT functions are called after each configuration takes place before any new requests are made. <tag/typedef void AUTHSPARSE(authScheme *, int, char *);/ Functions of type AUTHSPARSE are responsible for parsing authentication parameters. The function currently needs a scheme scope data structure to store the configuration in. The passed scheme's scheme_data pointer should point to the local data structure. Future development will allow all authentication schemes direct access to their configuration data without a locally scope structure. The parse function is called by Squid's config file parser when a auth_param scheme_name entry is encountered. <tag/typedef void AUTHSFREECONFIG(authScheme *);/ Functions of type AUTHSFREECONFIG are called by squid when freeing configuration data. The auth scheme should free any memory allocated that is related to parse data structures. The scheme MAY take advantage of this call to remove scheme local configuration dependent data. (Ie cached user details that are only relevant to a config setting). <tag/typedef void AUTHSDUMP(StoreEntry *, const char *, authScheme *);/ Functions of type AUTHSDUMP are responsible for writing to the StoreEntry the configuration parameters that a user would put in a config file to recreate the running configuration. <tag/typedef void AUTHSSTATS(StoreEntry *);/ Functions of type AUTHSSTATS are called by the cachemgr to provide statistics on the authmodule. Current modules simply provide the statistics from the back end helpers (number of requests, state of the helpers), but more detailed statistics are possible - for example unique users seen or failed authentication requests. The next set of functions work on the data structures used by the authentication schemes. <tag/typedef void AUTHSREQFREE(auth_user_request_t *);/ The AUTHSREQFREE function is called when a auth_user_request is being freed by the authentication framework, and scheme specific data was present. The function should free any scheme related data and MUST set the scheme_data pointer to NULL. Failure to unlink the scheme data will result in squid dying. <tag/typedef char *AUTHSUSERNAME(auth_user_t *);/ Squid does not make assumptions about where the username is stored. This function must return a pointer to a NULL terminated string to be used in logging the request. Return NULL if no username/usercode is known. The string should NOT be allocated each time this function is called. <tag/typedef int AUTHSAUTHED(auth_user_request_t *);/ The AUTHED function is used by squid to determine whether the auth scheme has successfully authenticated the user request. If timeouts on cached credentials have occurred or for any reason the credentials are not valid, return false.The next set of functions perform the actual authentication. The functions are used by squid for both WWW- and Proxy- authentication. Therefore they MUST NOT assume the authentication will be based on the Proxy-* Headers. <tag/typedef void AUTHSAUTHUSER(auth_user_request_t *, request_t *, ConnStateData *, http_hdr_type);/ Functions of type AUTHSAUTHUSER are called when Squid has a request that needs authentication. If needed the auth scheme can alter the auth_user pointer (usually to point to a previous instance of the user whose name is discovered late in the auth process. For an example of this see the NTLM scheme). These functions are responsible for performing any in-squid routines for the authentication of the user. The auth_user_request struct that is passed around is only persistent for the current request. If the auth module requires access to the structure in the future it MUST lock it, and implement some method for identifying it in the future. For example the NTLM module implements a connection based authentication scheme, so the auth_user_request struct gets referenced from the ConnStateData. <tag/typedef void AUTHSDECODE(auth_user_request_t *, const char *);/ Functions of type AUTHSDECODE are responsible for decoding the passed authentication header, creating or linking to a auth_user struct and for storing any needed details to complete authentication in AUTHSAUTHUSER. <tag/typedef int AUTHSDIRECTION(auth_user_request_t *);/ Functions of type AUTHSDIRECTION are used by squid to determine what the next step in performing authentication for a given scheme is. The following are the return codes: <itemize> <item>-2 = error in the auth module. Cannot determine request direction. <item>-1 = the auth module needs to send data to an external helper. Squid will prepare for a callback on the request and call the AUTHSSTART function. <item>0 = the auth module has all the information it needs to perform the authentication and provide a succeed/fail result. <item>1 = the auth module needs to send a new challenge to the request originator. Squid will return the appropriate status code (401 or 407) and call the registered FixError function to allow the auth module to insert it's challenge. </itemize> <tag/typedef void AUTHSFIXERR(auth_user_request_t *, HttpReply *, http_hdr_type, request_t *);/ Functions of type AUTHSFIXERR are used by squid to add scheme specific challenges when returning a 401 or 407 error code. On requests where no authentication information was provided, all registered auth modules will have their AUTHSFIXERR function called. When the client makes a request with an authentication header, on subsequent calls only the matching AUTHSFIXERR function is called (and then only if the auth module indicated it had a new challenge to send the client). If no auth schemes match the request, the authentication credentials in the request are ignored - and all auth modules are called. <tag/typedef void AUTHSFREE(auth_user_t *);/ These functions are responsible for freeing scheme specific data from the passed auth_user_t structure. This should only be called by squid when there are no outstanding requests linked to the auth user. This includes removing the user from any scheme specific memory caches. <tag/typedef void AUTHSADDHEADER(auth_user_request_t *, HttpReply *, int);/ <tag/typedef void AUTHSADDTRAILER(auth_user_request_t *, HttpReply *, int);/ These functions are responsible for adding any authentication specific header(s) or trailer(s) OTHER THAN the WWW-Authenticate and Proxy-Authenticate headers to the passed HttpReply. The int indicates whether the request was an accelerated request or a proxied request. For example operation see the digest auth scheme. (Digest uses a Authentication-Info header.) This function is called whenever a auth_user_request exists in a request when the reply is constructed after the body is sent on chunked replies respectively. <tag/typedef void AUTHSONCLOSEC(ConnStateData *);/ This function type is called when a auth_user_request is linked into a ConnStateData struct, and the connection is closed. If any scheme specific activities related to the request or connection are in progress, this function MUST clear them. <tag/typedef void AUTHSSTART(auth_user_request_t * , RH * , void *);/ This function type is called when squid is ready to put the request on hold and wait for a callback from the auth module when the auth module has performed it's external activities. </descrip> <sect2>Data Structures This is used to link auth_users into the username cache. Because some schemes may link in aliases to a user, the link is not part of the auth_user structure itself. <verb> struct _auth_user_hash_pointer { /* first two items must be same as hash_link */ char *key; auth_user_hash_pointer *next; auth_user_t *auth_user; dlink_node link; /* other hash entries that point to the same auth_user */ }; </verb> This is the main user related structure. It stores user-related data, and is persistent across requests. It can even persistent across multiple external authentications. One major benefit of preserving this structure is the cached ACL match results. This structure, is private to the authentication framework. <verb> struct _auth_user_t { /* extra fields for proxy_auth */ /* this determines what scheme owns the user data. */ auth_type_t auth_type; /* the index +1 in the authscheme_list to the authscheme entry */ int auth_module; /* we only have one username associated with a given auth_user struct */ auth_user_hash_pointer *usernamehash; /* we may have many proxy-authenticate strings that decode to the same user*/ dlink_list proxy_auth_list; dlink_list proxy_match_cache; struct { unsigned int credentials_ok:2; /*0=unchecked,1=ok,2=failed*/ } flags; long expiretime; /* IP addr this user authenticated from */ struct in_addr ipaddr; time_t ip_expiretime; /* how many references are outstanding to this instance*/ size_t references; /* the auth scheme has it's own private data area */ void *scheme_data; /* the auth_user_request structures that link to this. Yes it could be a splaytree * but how many requests will a single username have in parallel? */ dlink_list requests; }; </verb> This is a short lived structure is the visible aspect of the authentication framework. <verb> struct _auth_user_request_t { /* this is the object passed around by client_side and acl functions */ /* it has request specific data, and links to user specific data */ /* the user */ auth_user_t *auth_user; /* return a message on the 401/407 error pages */ char *message; /* any scheme specific request related data */ void *scheme_data; /* how many 'processes' are working on this data */ size_t references; }; </verb> The authscheme_entry struct is used to store the runtime registered functions that make up an auth scheme. An auth scheme module MUST implement ALL functions except the following functions: oncloseconnection, AddHeader, AddTrailer.. In the future more optional functions may be added to this data type. <verb> struct _authscheme_entry { char *typestr; AUTHSACTIVE *Active; AUTHSADDHEADER *AddHeader; AUTHSADDTRAILER *AddTrailer; AUTHSAUTHED *authenticated; AUTHSAUTHUSER *authAuthenticate; AUTHSDUMP *dump; AUTHSFIXERR *authFixHeader; AUTHSFREE *FreeUser; AUTHSFREECONFIG *freeconfig; AUTHSUSERNAME *authUserUsername; AUTHSONCLOSEC *oncloseconnection; /*optional*/ AUTHSDECODE *decodeauth; AUTHSDIRECTION *getdirection; AUTHSPARSE *parse; AUTHSINIT *init; AUTHSREQFREE *requestFree; AUTHSSHUTDOWN *donefunc; AUTHSSTART *authStart; AUTHSSTATS *authStats; }; </verb> For information on the requirements for each of the functions, see the details under the typedefs above. For reference implementations, see the squid source code, /src/auth/basic for a request based stateless auth module, and /src/auth/ntlm for a connection based stateful auth module. <sect2>How to add a new Authentication Scheme Copy the nearest existing auth scheme and modify to receive the appropriate scheme headers. Now step through the acl.c MatchAclProxyUser function's code path and see how the functions call down through authenticate.c to your scheme. Write a helper to provide you scheme with any backend existence it needs. Remember any blocking code must go in AUTHSSTART function(s) and _MUST_ use callbacks. <sect2>How to ``hook in'' new functions to the API Start of by figuring the code path that will result in the function being called, and what data it will need. Then create a typedef for the function, add and entry to the authscheme_entry struct. Add a wrapper function to authenticate.c (or if appropriate cf_cache.c) that called the scheme specific function if it exists. Test it. Test it again. Now port to all the existing auth schemes, or at least add a setting of NULL for the function for each scheme. <sect1>Authentication Module Interface <sect2>Basic Authentication Modules Basic authentication provides a username and password. These are written to the authentication module processes on a single line, separated by a space: <verb> <USERNAME> <PASSWORD> </verb> The authentication module process reads username, password pairs on stdin and returns either ``OK'' or ``ERR'' on stdout for each input line. The following simple perl script demonstrates how the authentication module works. This script allows any user named ``Dirk'' (without checking the password) and allows any user that uses the password ``Sekrit'': <verb> #!/usr/bin/perl -w $|=1; # no buffering, important! while (<>) { chop; ($u,$p) = split; $ans = &check($u,$p); print "$ans\n"; } sub check { local($u,$p) = @_; return 'ERR' unless (defined $p && defined $u); return 'OK' if ('Dirk' eq $u); return 'OK' if ('Sekrit' eq $p); return 'ERR'; } </verb>  <sect>ICP To be written...  <sect>Network Measurement Database To be written...  <sect>Error Pages To be written...  <sect>Callback Data Allocator Squid's extensive use of callback functions makes it very susceptible to memory access errors. For a blocking operation with callback functions, the normal sequence of events is as follows: <verb> callback_data = malloc(...); ... fooOperationStart(bar, callback_func, callback_data); ... fooOperationComplete(...); callback_func(callback_data, ....); ... free(callback_data); </verb> However, things become more interesting if we want or need to free the callback_data, or otherwise cancel the callback, before the operation completes. The callback data allocator lets us do this in a uniform and safe manner. The callback data allocator is used to allocate, track and free memory pool objects used during callback operations. Allocated memory is locked while the blocking operation executes elsewhere, and is freed when the operation completes. The normal sequence of events is: <verb> type_of_data callback_data; ... callback_data = cbdataAlloc(type_of_data); ... cbdataLock(callback_data); fooOperationStart(bar, callback_func, callback_data); ... fooOperationComplete(...); if (cbdataValid(callback_data)) { callback_func(callback_data, ....); cbdataUnlock(callback_data); cbdataFree(callback_data); </verb> With this scheme, nothing bad happens if <tt/cbdataFree/ gets called before <tt/cbdataUnlock/: <verb> callback_data = cbdataAlloc(...); ... cbdataLock(callback_data); fooOperationStart(bar, callback_func, callback_data); ... cbdataFree(callback_data); ... fooOperationComplete(...); if (cbdataValid(callback_data)) { callback_func(callback_data, ....); cbdataUnlock(callback_data); </verb> In this case, when <tt/cbdataFree/ is called before <tt/cbdataUnlock/, the callback_data gets marked as invalid. Before executing the callback function, <tt/cbdataValid/ will return 0 and callback_func is never executed. When <tt/cbdataUnlock/ gets called, it notices that the callback_data is invalid and will then call <tt/cbdataFree/. To add new module specific data types to the allocator one uses the macros CBDATA_TYPE and CBDATA_INIT_TYPE. These creates a local cbdata definition (file or block scope). Any cbdataAlloc calls must be made within this scope. However, cbdataFree might be called from anywhere. <verb> /* First the cbdata type needs to be defined in the module. This * is usually done at file scope, but it can also be local to a * function or block.. */ CBDATA_TYPE(type_of_data); /* Then in the code somewhere before the first allocation * (can be called multiple times with only a minimal overhead) */ CBDATA_INIT_TYPE(type_of_data); /* Or if a free function is associated with the data type */ CBDATA_INIT_TYPE_FREECB(type_of_data, free_function); </verb> To add new global data types one have to add them to the cbdata_type enum in enums.h, and a corresponding CREATE_CBDATA call in cbdata.c:cbdataInit(). Or alternatively add a CBDATA_GLOBAL_TYPE definition to globals.h and use CBDATA_INIT_TYPE as described above. <verb> extern CBDATA_GLOBAL_TYPE(type_of_data); /* CBDATA_UNDEF */ </verb>  <sect>Cache Manager To be written...  <sect>HTTP Headers <em/Files:/ <tt/HttpHeader.c/, <tt/HttpHeaderTools.c/, <tt/HttpHdrCc.c/, <tt/HttpHdrContRange.c/, <tt/HttpHdrExtField.c/, <tt/HttpHdrRange.c/ <tt/HttpHeader/ class encapsulates methods and data for HTTP header manipulation. <tt/HttpHeader/ can be viewed as a collection of HTTP header-fields with such common operations as add, delete, and find. Compared to an ascii "string" representation, <tt/HttpHeader/ performs those operations without rebuilding the underlying structures from scratch or searching through the entire "string". <sect1>General remarks <tt/HttpHeader/ is a collection (or array) of HTTP header-fields. A header field is represented by an <tt/HttpHeaderEntry/ object. <tt/HttpHeaderEntry/ is an (id, name, value) triplet. Meaningful "Id"s are defined for "well-known" header-fields like "Connection" or "Content-Length". When Squid fails to recognize a field, it uses special "id", <em/HDR_OTHER/. Ids are formed by capitalizing the corresponding HTTP header-field name and replacing dashes ('-') with underscores ('_'). Most operations on <tt/HttpHeader/ require a "known" id as a parameter. The rationale behind the later restriction is that Squid programmer should operate on "known" fields only. If a new field is being added to header processing, it must be given an id. <sect1>Life cycle <tt/HttpHeader/ follows a common pattern for object initialization and cleaning: <verb> /* declare */ HttpHeader hdr; /* initialize (as an HTTP Request header) */ httpHeaderInit(&hdr, hoRequest); /* do something */ ... /* cleanup */ httpHeaderClean(&hdr); </verb> Prior to use, an <tt/HttpHeader/ must be initialized. A programmer must specify if a header belongs to a request or reply message. The "ownership" information is used mostly for statistical purposes. Once initialized, the <tt/HttpHeader/ object <em/must/ be, eventually, cleaned. Failure to do so will result in a memory leak. Note that there are no methods for "creating" or "destroying" a "dynamic" <tt/HttpHeader/ object. Looks like headers are always stored as a part of another object or as a temporary variable. Thus, dynamic allocation of headers is not needed. <sect1>Header Manipulation The mostly common operations on HTTP headers are testing for a particular header-field (<tt/httpHeaderHas()/), extracting field-values (<tt/httpHeaderGet*()/), and adding new fields (<tt/httpHeaderPut*()/). <tt/httpHeaderHas(hdr, id)/ returns true if at least one header-field specified by "id" is present in the header. Note that using <em/HDR_OTHER/ as an id is prohibited. There is usually no reason to know if there are "other" header-fields in a header. <tt/httpHeaderGet<Type>(hdr, id)/ returns the value of the specified header-field. The "Type" must match header-field type. If a header is not present a "null" value is returned. "Null" values depend on field-type, of course. Special care must be taken when several header-fields with the same id are preset in the header. If HTTP protocol allows only one copy of the specified field per header (e.g. "Content-Length"), <tt/httpHeaderGet<Type>()/ will return one of the field-values (chosen semi-randomly). If HTTP protocol allows for several values (e.g. "Accept"), a "String List" will be returned. It is prohibited to ask for a List of values when only one value is permitted, and visa-versa. This restriction prevents a programmer from processing one value of an header-field while ignoring other valid values. <tt/httpHeaderPut<Type>(hdr, id, value)/ will add an header-field with a specified field-name (based on "id") and field_value. The location of the newly added field in the header array is undefined, but it is guaranteed to be after all fields with the same "id" if any. Note that old header-fields with the same id (if any) are not altered in any way. The value being put using one of the <tt/httpHeaderPut()/ methods is converted to and stored as a String object. Example: <verb> /* add our own Age field if none was added before */ int age = ... if (!httpHeaderHas(hdr, HDR_AGE)) httpHeaderPutInt(hdr, HDR_AGE, age); </verb> There are two ways to delete a field from a header. To delete a "known" field (a field with "id" other than <em/HDR_OTHER/), use <tt/httpHeaderDelById()/ function. Sometimes, it is convenient to delete all fields with a given name ("known" or not) using <tt/httpHeaderDelByName()/ method. Both methods will delete <em/all/ fields specified. The <em/httpHeaderGetEntry(hdr, pos)/ function can be used for iterating through all fields in a given header. Iteration is controlled by the <em/pos/ parameter. Thus, several concurrent iterations over one <em/hdr/ are possible. It is also safe to delete/add fields from/to <em/hdr/ while iteration is in progress. <verb> /* delete all fields with a given name */ HttpHeaderPos pos = HttpHeaderInitPos; HttpHeaderEntry *e; while ((e = httpHeaderGetEntry(hdr, &pos))) { if (!strCaseCmp(e->name, name)) ... /* delete entry */ } </verb> Note that <em/httpHeaderGetEntry()/ is a low level function and must not be used if high level alternatives are available. For example, to delete an entry with a given name, use the <em/httpHeaderDelByName()/ function rather than the loop above. <sect1>I/O and Headers To store a header in a file or socket, pack it using <tt/httpHeaderPackInto()/ method and a corresponding "Packer". Note that <tt/httpHeaderPackInto/ will pack only header-fields; request-lines and status-lines are not prepended, and CRLF is not appended. Remember that neither of them is a part of HTTP message header as defined by the HTTP protocol. <sect1>Adding new header-field ids Adding new ids is simple. First add new HDR_ entry to the http_hdr_type enumeration in enums.h. Then describe a new header-field attributes in the HeadersAttrs array located in <tt/HttpHeader.c/. The last attribute specifies field type. Five types are supported: integer (<em/ftInt/), string (<em/ftStr/), date in RFC 1123 format (<em/ftDate_1123/), cache control field (<em/ftPCc/), range field (<em/ftPRange/), and content range field (<em/ftPContRange/). Squid uses type information to convert internal binary representation of fields to their string representation (<tt/httpHeaderPut/ functions) and visa-versa (<tt/httpHeaderGet/ functions). Finally, add new id to one of the following arrays: <em/GeneralHeadersArr/, <em/EntityHeadersArr/, <em/ReplyHeadersArr/, <em/RequestHeadersArr/. Use HTTP specs to determine the applicable array. If your header-field is an "extension-header", its place is in <em/ReplyHeadersArr/ and/or in <em/RequestHeadersArr/. You can also use <em/EntityHeadersArr/ for "extension-header"s that can be used both in replies and requests. Header fields other than "extension-header"s must go to one and only one of the arrays mentioned above. Also, if the new field is a "list" header, add it to the <em/ListHeadersArr/ array. A "list" field-header is the one that is defined (or can be defined) using "&num;" BNF construct described in the HTTP specs. Essentially, a field that may have more than one valid field-value in a single header is a "list" field. In most cases, if you forget to include a new field id in one of the required arrays, you will get a run-time assertion. For rarely used fields, however, it may take a long time for an assertion to be triggered. There is virtually no limit on the number of fields supported by Squid. If current mask sizes cannot fit all the ids (you will get an assertion if that happens), simply enlarge HttpHeaderMask type in <tt/typedefs.h/. <sect1>A Word on Efficiency <tt/httpHeaderHas()/ is a very cheap (fast) operation implemented using a bit mask lookup. Adding new fields is somewhat expensive if they require complex conversions to a string. Deleting existing fields requires scan of all the entries and comparing their "id"s (faster) or "names" (slower) with the one specified for deletion. Most of the operations are faster than their "ascii string" equivalents. <sect>File Formats <sect1><em/swap.state/ NOTE: this information is current as of version 2.2.STABLE4. A <em/swap.state/ entry is defined by the <em/storeSwapLogData/ structure, and has the following elements: <verb> struct _storeSwapLogData { char op; int swap_file_number; time_t timestamp; time_t lastref; time_t expires; time_t lastmod; size_t swap_file_sz; u_short refcount; u_short flags; unsigned char key[MD5_DIGEST_CHARS]; }; </verb> <descrip> <tag/op/ Either SWAP_LOG_ADD (1) when an object is added to the disk storage, or SWAP_LOG_DEL (2) when an object is deleted. <tag/swap_file_number/ The 32-bit file number which maps to a pathname. Only the low 24-bits are relevant. The high 8-bits are used as an index to an array of storage directories, and are set at run time because the order of storage directories may change over time. <tag/timestamp/ A 32-bit Unix time value that represents the time when the origin server generated this response. If the response has a valid <em/Date:/ header, this timestamp corresponds to that time. Otherwise, it is set to the Squid process time when the response is read (as soon as the end of headers are found). <tag/lastref/ The last time that a client requested this object. Strictly speaking, this time is set whenver the StoreEntry is locked (via <em/storeLockObject()/). <tag/expires/ The value of the response's <em/Expires:/ header, if any. If the response does not have an <em/Expires:/ header, this is set to -1. If the response has an invalid (unparseable) <em/Expires:/ header, it is also set to -1. There are some cases where Squid sets <em/expires/ to -2. This happens for the internal ``netdb'' object and for FTP URL responses. <tag/lastmod/ The value of the response's <em/Last-modified:/ header, if any. This is set to -1 if there is no <em/Last-modified:/ header, or if it is unparseable. <tag/swap_file_sz/ This is the number of bytes that the object occupies on disk. It includes the Squid ``swap file header''. <tag/refcount/ The number of times that this object has been accessed (referenced). Since its a 16-bit quantity, it is susceptible to overflow if a single object is accessed 65,536 times before being replaced. <tag/flags/ A copy of the <em/StoreEntry/ flags field. Used as a sanity check when rebuilding the cache at startup. Objects that have the KEY_PRIVATE flag set are not added back to the cache. <tag/key/ The 128-bit MD5 hash for this object. </descrip> Note that <em/storeSwapLogData/ entries are written in native machine byte order. They are not necessarily portable across architectures. <sect>Store ``swap meta'' Description ``swap meta'' refers to a section of meta data stored at the beginning of an object that is stored on disk. This meta data includes information such as the object's cache key (MD5), URL, and part of the StoreEntry structure. The meta data is stored using