Squid Proxy Cache
NOTE: The information on this page is offered “as is” - we cannot provide support beyond what is written here.
From the squid web page: Squid is a caching proxy for the Web supporting HTTP, HTTPS, FTP, and more. It reduces bandwidth and improves response times by caching and reusing frequently-requested web pages. Squid has extensive access controls and makes a great server accelerator. It runs on most available operating systems, including Windows and is licensed under the GNU GPL.
Squid can be used to have a local texture cache bigger than the viewer maximum. Use a spare 500GB drive or even a 1TB drive to cache all those SL textures for reduced bandwidth and speed up rezzing. The cache can be shared by the whole house. No more downloading the same texture for each SL user in the house.
Install or make sure Perl is working, many Linux distributions come with Perl installed. http://www.perl.org/ or the support site of your Linux install. http://www.perl.org/get.html has links for a windows version of Perl available.
Many Linux distributions come with squid proxy available by the package manager. Consult the support site for that Linux distribution (distro for lazy people) for more info.
Install squid proxy version 2.7 from http://www.squid-cache.org/ Windows binary available at http://squid.acmeconsulting.it/ Squid proxy version 3 will not work, it is missing a feature that is required to work with HTTP Get textures. There is plenty of documentation on the squid web site and tutorials around the net on how to install squid. There is even a book on it.
Once you have squid installed or unpacked, and tested with a browser that it is working. Shut it down in order to make the following changes.
Create a file somewhere on the system, I used /etc/squid/store_url_rewrite Put in that file the following:
- store_url_rewrite
#!/usr/bin/perl $|=1; while(<>){chomp;if(m/sim(.*?)\.agni\.lindenlab\.com(.*?)\/cap\/(.*?)\/\?texture_id=(.*)/){print"http://agni.lindenlab.com/texture/".$4."\n";goto STOP;}if(m/sim(.*?)\.agni\.lindenlab\.com(.*?)\/cap\/(.*?)\/\?mesh_id=(.*)/){print"http://agni.lindenlab.com/mesh/".$4."\n";goto STOP;}if(m/asset-cdn\.agni\.lindenlab\.com(.*?)\/\?texture_id=(.*)/){print"http://agni.lindenlab.com/texture/".$2."\n";goto STOP;}if(m/asset-cdn\.agni\.lindenlab\.com(.*?)\/\?mesh_id=(.*)/){print"http://agni.lindenlab.com/mesh/".$2."\n";goto STOP;}print$_."\n"; STOP:}
- For Windows users: Save the code above into a file called “store_url_rewrite.pl” and move it to C:\squid\etc\
- For Linux users: Save the file and then make it executable with “chmod +x /etc/squid/store_url_rewrite”
Wanted: A python version of this script and other languages. Contact Liny Odell if you wish to submit a script in a different language.
Navigate to the directory you installed or unpacked squid to, example: C:\squid\etc and duplicate mime.conf.default and squid.conf.default.
Then rename these 2 files so their ending is “conf”.
Now to make the following config changes in squid.conf:
Search for the following lines and place a “#” in front of them to disable the config lines.
http_access deny !Safe_ports http_access deny CONNECT !SSL_ports
The should look like this or completly removed.
#http_access deny !Safe_ports #http_access deny CONNECT !SSL_ports
If you tested squid with your browser, and received an access denied error message, then you may have to add another line to allow localhost.
See the last line in the following section:
# Example rule allowing access from your local networks. # Adapt localnet in the ACL section to list your (internal) IP networks # from where browsing should be allowed http_access allow localnet http_access allow localhost #this line was required for me to get it running
Look for the refresh_pattern section and add “refresh_pattern /cap/ 259200 20% 302400” as the first one, like this:
refresh_pattern /cap/ 259200 20% 302400 refresh_pattern asset-cdn\.agni\.lindenlab\.com/.* 259200 20% 302400 refresh_pattern ^ftp: 1440 20% 10080 refresh_pattern ^gopher: 1440 0% 1440 refresh_pattern -i (/cgi-bin/|\?) 0 0% 0
The refresh line is set up to cache texture for six months. Due to LL not using Last Modify: headers, there no way for squid to tell if the texture has been changed or not. Since LL doesn't change textures at all, store them for however long is needed. The “20%” and “302400” have no effect due to when squid check freshness on a file, it has to re-download the entire file. Blame LL for this. The “302400” need to be greater then the first column, “259200”
- For Windows users:
Copy & paste the lines below to the end of your squid.conf file, and modify paths to resemble those on your system accordingly:
# 64 GB cache size, can make larger later. cache_dir ufs c:/squid/var/cache 64000 16 256 # disable range requests due to LL fail. No Last Modify header. range_offset_limit -1 # this is a local cache, prevent possible error visible_hostname MyBigCache # this is the magic, convert the URL into something more static. storeurl_rewrite_program C:/strawberry-perl/perl/bin/perl.exe c:/squid/etc/store_url_rewrite.pl # increase the children to handle the default Phoenix Viewer client of 32 HTTP texture fetches in the queue. storeurl_rewrite_children 32 # what domains to apply the store url rewrite too. acl store_rewrite_list dstdomain .agni.lindenlab.com storeurl_access allow store_rewrite_list storeurl_access deny all # dont tell http-in we are proxying or it wont work right header_access Via deny all header_access Forwarded-For deny all header_access X-Forwarded-For deny all #dont cache baked textures acl bakeserver dstdomain bake-texture.agni.lindenlab.com cache deny bakeserver
- For Linux users:
Add the following lines to squid.conf
# 64 GB cache size, can make larger later. cache_dir ufs /var/cache/squid 64000 16 256 # disable range requests due to LL fail. No Last Modify header. range_offset_limit -1 # this is a local cache, prevent possible error visible_hostname MyBigCache # this is the magic, convert the URL into something more static. storeurl_rewrite_program /etc/squid/store_url_rewrite # increase the children to handle the default Phoenix Viewer client of 32 http texture fetches in the queue. storeurl_rewrite_children 32 # what domains to apply the store url rewrite too. acl store_rewrite_list dstdomain .agni.lindenlab.com storeurl_access allow store_rewrite_list storeurl_access deny all # dont tell http-in we are proxying or it wont work right header_access Via deny all header_access Forwarded-For deny all header_access X-Forwarded-For deny all #dont cache baked textures acl bakeserver dstdomain bake-texture.agni.lindenlab.com cache deny bakeserver
After saving squid.conf, on Windows systems, open a cmd window, and run this command to initialize your previously defined cache directory:
C:\squid\sbin\squid.exe -z
Start up squid proxy (c:\squid\sbin\squid.exe on Windows).
- Start up Firestorm.
- Ctrl-p to bring up Preferences.
- Under the Network & Files -> Connection tab, click on the Adjust proxy settings button; this opens the window shown below.
- Enable Use HTTP Proxy for Web Pages
- Add the hostname or ip-address of the squid proxy to the HTTP Proxy field; if on the same computer, add “localhost”.
- Set Port number to “3128”.
- Under Other HTTP traffic proxy, set to Use HTTP Proxy.
- Click “Ok” to save and close the preference window.
- Click “Ok” and restart the viewer.
After starting the viewer back up, you should see the login splash page. If not, re-check the setting and make sure squid proxy and perl are working. You can test this by using a normal browser configured to use the proxy.
After logging back into world, you can enable HTTP Get Textures by going to Preferences → Graphics -> Rendering→ HTTP Get textures.
See this page for documentation on Firestorm 4.6.9 (42969) and earlier.