Apache Proxy Security Issue

Recently We were deploying a ROR (Ruby On Rails) application to be specific its Redmine. Since our web server had too many virtual host running on Apache we couldn't run the webrick web server directly on port 80. We decided to run it on Port 8000 and let apache virtual host for this redmine will be proxying to the port 8000

Whats the configutration?

-------------- Configurations Begins ---------------
<VirtualHost...... >
ProxyRequests On
ProxyVia On


....

ProxyPass / http://localhost:8000/
ProxyPassReverse / http://localhost:8000/

.....
</VirtualHost.>
-------------- Configurations Ends -----------------

Later a month we observed our web server became too slow, We saw the response taking too much time. Looking at the performance Apache was consuming more memory and cpu load.

Just a top command explained the change in apache's behaviour

Looking into Apache's access log we saw too many web requests unrelated domains to the server were accessed. Finally we realised that the apache became a proxy server and now it is acting as a proxy to many people and they access their banned sites through the apache's proxy service.

The fix is to remove the following entries

ProxyRequests On
ProxyVia On

The ROR application was still proxied because of the other entry in the VirtualHost. The application still worked and we stopped the open proxy behaviour.

Once the fix was done, we observed all the proxy request in the access logs were denied with 404 and thus the server is saved ;-)


SQUID Load balancing for web applications

Squid Load Balancer Configurations

A short note on we have done to make the squid load balancer working
The servers are CEnt OS 5.x

Total machines 2 (Don't ask me why it is 2, This is what I had to test and play with in my lab)
Machine 1 - IPs 192.168.5.50 and 192.168.5.51 (2 LAN cards)
Machine 2 - IPs 192.168.100 and 192.168.5.101 (2 LAN cards)

The machine 1 and 2 Will have Apache application running on them which needs to be load balanced.

Problem: We need a seperate machine which should act as load balancer for both, but we don't have any other machine than these 2.
Solution: Machine 1 will act as load balancer and also as a application server(not adviceable) but can go ahead if we run out of resource as we have no other option.

DNS Names
DOMAIN Name for public access webapp.office.lan

webapp.office.lan - 192.168.5.50
server_1_a.office.lan - 192.168.5.50
server_1_b.office.lan - 192.168.5.51

server_2_a.office.lan - 192.168.5.100
server_2_b.office.lan - 192.168.5.101

The web application is configured in both the server with apache vhosts.


The default site is webapp.office.lan and squid listens to the queries on that.

So the first part is making the application work in

server_1_b.office.lan - 192.168.5.51
server_2_b.office.lan - 192.168.5.101

Configuring apache right to make this work properly.
Since Squid and Apache are going to run on same machine(1) and are to be used in same port number 80 we need to make apache listen only for requests on IP:192.168.5.51

To do the above
Change the httpd.conf
Modify the line
Listen 80
to
Listen 192.168.5.51:80

So apache listens only the IP: 192.168.5.51 on Port 80

Now configure the vhost of the web app accordingly in Apache on IP:192.168.5.51
Verify do the site works by accessing server_1_b.office.lan

Configuring App in server_2_b.office.lan - 192.168.5.101
Since the app is now configured on a different machine we don't need to change the Apache Listen property.
but once configured just check the site works with server_2_b.office.lan URL

Squid in Action
Installing squid in CentOS is as same as installing Apache with yum installer.

Configuring Squid
Add the following lines in /etc/squid/squid.conf

------------------------------ Lines to be added in squid.conf -------------------------------
#Make SQUID Listen on PORT 80
http_port 192.168.5.50:80 defaultsite=webapp.office.lan vhost

# Mapping 192.168.5.51 as server_1
cache_peer server_1_b.office.lan parent 80 0 no-query originserver name=server_1 login=PASS
cache_peer_domain server_1 server_1_b.office.lan login=PASS

# Mapping 192.168.5.101 as server_2
cache_peer server_2_b.office.lan parent 80 0 no-query originserver name=server_2 login=PASS
cache_peer_domain server_2 server_2_b.office.lan login=PASS

cache_peer server_1_b.office.lan parent 80 0 round-robin no-query originserver login=PASS
cache_peer server_2_b.office.lan parent 80 0 round-robin no-query originserver login=PASS

----------------------------- End of lines to be added in squid.conf ----------------------


Now the squid can be restarted.
The server will be listening to webapp.office.lan each request will be diverted to different server based on the round robin flow and if any one fails the other will server the request continuously we can add 'n' servers similar to this to make the count higher
Note all the vhost in the Apache should listen to domain name "webapp.office.lan".

Hope this gave a useful information on SQUID loadbalancing, This is not only for apache but can be any webserver serving similarly.

SQUID Load Balancing For HTTP-AUTH Applications

Recently I was working with SQUID Load Balancing server for one of the PHP Based Web Application.

The app uses HTTP-AUTH for one of its protected directory. It uses Apache .htaccess with .htpasswd Unfortunately the login was completely failing in the live environment but not in test environment.

The difference found was the live environment had a SQUID Load balancing which was not in TEST (Something wrong should not be the case, both environments should resemble similar).

Then it was observed that the User name and password sent from the client is not reaching the real-application server. It is chopped at the SQUID.

Why SQUID is not passing the information?
SQUID has features to do proxy / load balancing with authentication, where SQUID assumes that the AUTH header is for SQUID and not for the web application so it never forwards the AUTH Header.

How to forward the AUTH Header?
Looking in to the squid.conf

cache_peer IP.ADDRESS parent 80 0 no-query originserver login=PASS

The last suffix login=PASS fixed the problem.

The login=PASS forwards the HTTP-AUTH credentials to the destination server.

SQUID Clearing cache in a load balancer / caching server

Are you running a SQUID Caching server before your web server to boost up the performance?
If yes and you face issues when the content of the site changes. It might be due to the squid's cache still having the old content. The following steps will help to refresh the cache.

Why we need to clear the cache?
In most cases the content in the cache is out dated with the live data.

What is the normal way to clear the cache?
We can clear the cache by removing the files in the cache directory.

Where is the cache directory?
The cache directory changes from system to system based on the configuration file settings.
We can find the cache directory by looking for the cache_dir property in the /etc/squid/squid.conf file.
Steps to clear the cache:
1. Login as privileged user.
2. Shutdown squid.
Eg: Fedora / Redhat / CentOS
# service squid stop
3. Remove the cache files.
The directory is the path specified in cache_dir
#rm -rf /var/spool/squid/*
4. Start the squid again
# service squid start
5. We should be able to see a message cache created in cache_dir directory.

Make Dynamic VirtualHost in Apache

Are you working in Apache? Are you configuring VritualHost often?
Here is a cool solution that avoids us configuring the VirtualHost directive in Apache often.
Any name based apache virtual host will be automatically mapped with some predefined directory path. Which reduces the time of configuring the apache vhosts.

The below example config change in apache will do the following

1. Configures all virtual host to the /srv/www directory
2. All VirtualHost by name should have a directory with the domain name. Ex: Vhost test.example.com will have a directory /srv/www/test.example.com
3. The Document root directory will be htdocs by default
4. The error log will be added to the specific domain file and all access logs are added to common file.
   

Example Configuration to add in httpd.conf
# this log format can be split per-virtual-host based on the first field
LogFormat "%V %h %l %u %t \"%r\" %s %b" vcommon
CustomLog logs/access_log vcommon
#ErrorLog logs/%0_error_log

# include the server name in the filenames used to satisfy requests
VirtualDocumentRoot /srv/www/%0/htdocs
VirtualScriptAlias /srv/www/%0/cgi-bin

Easy SVN Web Administration

In the last month blog we were looking into "How to install and configure an SVN server".
The blog gives a basic SVN server configuration with command line and managing it through the command line. When we end up with more projects / users we need more repositories and managing the users authentication details becomes a nightmare. To simplify the user creation and repository management we can go for a web based SVN solution.

;-) Ohh don't think that we are going to do the Apache setup for each repository and more. We have a better solution.

Here comes a better SVN with USVN (User friendly SVN)

What do we need for this?
Requirements:

• PHP 5 (5.1.2 <= ver)
• apache2
• mod_dav_svn enable
• mod_rewrite enable
• subversion
• mod_svn enable
• mod_authz_svn enable

Steps to setup.
1. Download USVN from http://www.usvn.info/download
2. Extract the zip to the root directory of apache web directory
3. Access the page via web to start the installation and proceed through the installation.

Sample configurations followed in Success Factory.
1. Apache Config

###########################################
# Vhost: svn.successfactory.local #
# Note: we need proper DNS setup to #
# make the URL work #
###########################################
<VirtualHost *:80>
DocumentRoot /srv/www/svn.successfactory.local/htdocs
ServerName svn.successfactory.local
<Directory />
AllowOverride All
</Directory>
<Location /repository/>
ErrorDocument 404 default
DAV svn
Require valid-user
SVNParentPath /srv/svn
SVNListParentPath off
AuthType Basic
AuthName "USVN"
AuthUserFile /srv/svn/htpasswd
AuthzSVNAccessFile /srv/svn/authz
</Location>
</VirtualHost>

The SVN repository resides in /srv/svn as explained in SVN installation post.
An SQLite DB is selected to maintain the SVN management informations.

After installation we can manage the SVN through web like
http://svn.successfactory.local

You will come across a login page as shown below.




We can manage new projects / users / groups through the simple web panel as shown below.





Hope this USVN brings a peace of mind in administring multiple SVN repositories.

Apache RewriteMap with RewriteLock

Recently I was working with a image gallery site. The site was developed with PHP on Apache which stores images and renders it out.

The photos where stored in a similar path stated below.
/images/photo_id/photos_style/photo_id.jpg

Example:
/images/200/portrait/200.jpg

The requirement was not to show the original path in URL and it should resemble like the below
/<photo_id>/<photo_id>_<_style>.jpg

Example:
/200/200_portrait.jpg

The logic had more complexity than explained here which required a math calculation to get the complete path. To achieve the calculation a perl rewrite rule was introduced.

RewriteMap prg MapType:/path/to/rewrite_rule.pl

The perl script was something similar to below with more logic

#!/usr/bin/perl
$| = 1;
while () {
    # ...put here any transformations or lookups...
    print $_;
}

The script started working well by redirecting to original directory (internally) with output like
/images/200/portrait/200.jpg

But when the server got loaded heavily with more requests. The output scrambled like
/mages/200/portit/200.jpg
/ramages/200/portrait/200.jpg

etc...

Which was due to the perl script not in sync with Apache. The problem was solved when a RewriteLock was introduced. But still a surprise how this solved it immediately... ;-)

RewriteLock "/path/to/empty/lock/file"
in the global section of httpd.conf