Post Reply  Post Thread 
Nonprofit requesting ideas for busy server
kevin
Junior Member
**

Posts: 3
Group: Registered
Joined: Jun 2008
Status: Offline
Reputation: 0
Post: #1
Nonprofit requesting ideas for busy server

Hello, we are a popular nonprofit organization who is experiencing growing pains. I am the volunteer sysadmin and still consider myself a linux newbie but have a few years under my belt. We do not have a budget to change any hardware spec at this time, so we are constantly trying to optimize and make what we have work.

Our organization is growing fast and we currently have over 35,000 visitors per day and typically 700-900 users online simultaneously during peak usage. After about 900 users we start to see some painful drag on the system.

I've done various adjustments with good success over the years, but I'm not sure if the current configs are fully optimal yet. We're now looking at implementing "curbs" on expensive operations that would essentially monitor the server health and restrict certain features from escalating the problem during times of heavy load.

My belief is that the #1 problem is disk IO, and #2 would be the MySQL and possibly Apache configs. We continue to optimize our SQL queries but we want to ensure that the system-level configs are as optimized as possible.

According to some of the articles I've read here, this server is EXTREMELY overloaded (you will be shocked when you see the metrics). However, we are okay with that as long as primary functions on the system are working correctly.

This is a social networking organization that provides regional community websites. We offload any tasks where we can find a reasonable third party (for example inbound mail was recently moved to Google Apps). We currently operate about 95 (virtually hosted) domains.

The server is a dedicated dual AMD Opteron 244 with 4 gigs RAM and RAID5 running Ubuntu Linux Feisty (7.04).

What information could I provide here that would be a good place to start with to help get some advice?

Thank you in advance.
Kevin

06-11-2008 03:21 PM
Find all posts by this user Quote this message in a reply
Pacifier
Administrator
*******

Posts: 214
Group: Administrators
Joined: May 2006
Status: Offline
Reputation: 0
Post: #2
RE: Nonprofit requesting ideas for busy server

All these will be helpful:

1) Output of:
* free -m
* top

2) Data of:
* my.cnf
* httpd.conf

3) Do you have access logs enabled on apache?
4) What's the language used mostly on the backend? If it's PHP, do you have mod_php or CGI/FastCGI?

06-11-2008 04:22 PM
Find all posts by this user Quote this message in a reply
kevin
Junior Member
**

Posts: 3
Group: Registered
Joined: Jun 2008
Status: Offline
Reputation: 0
Post: #3
RE: Nonprofit requesting ideas for busy server

Thanks for your reply. I will try to answer all your questions here:

free -m
About 571 users online now, so system is snappy. (I added dots so headers would stay lined up)

Code:
.............total       used       free     shared    buffers     cached
Mem:          3995       3537        458          0         51       1452
-/+ buffers/cache:       2033       1962
Swap:         9318         22       9296


top

Code:
top - 17:18:09 up 137 days,  3:59,  1 user,  load average: 14.68, 14.63, 14.75
Tasks: 157 total,  26 running, 121 sleeping,   0 stopped,  10 zombie
Cpu(s): 73.1%us, 25.1%sy,  0.0%ni,  0.0%id,  0.4%wa,  0.1%hi,  1.4%si,  0.0%st
Mem:   4091784k total,  3674332k used,   417452k free,    53380k buffers
Swap:  9542568k total,    23056k used,  9519512k free,  1509044k cached


my.cnf
(These are overrides from the default Ubuntu my.cnf)

Code:
# new config 080319
# based on http://www.vbulletin.com/forum/showthread.php?t=179439
# and also comments from percona presentations http://www.mysqlperformanceblog.com/mysql-performance-presentations/

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld]
bind-address = 127.0.0.1
user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
language = /usr/share/mysql/english
safe-show-database
skip-bdb
skip-innodb
# increased from 650 to 850 on 5/22 after hitting 1040 errors and per http://dev.mysql.com/doc/refman/5.0/en/too-many-connections.html
# increased from 850 to 1000 on 5/28 based on improvements seen when system is busy
max_connections = 1000
# percona recommends key_buffer = 30% of memory for MyISAM-only systems
key_buffer = 1228M
myisam_sort_buffer_size = 64M
join_buffer_size = 1M
# percona recommends to start read_buffer_size=1M, read_rnd_buffer_size=4M
read_buffer_size = 1M
read_rnd_buffer_size = 4M
sort_buffer_size = 2M
#080220 -- increased table_cache from 1800 because opened tables = 525 k
#080319 -- decreased table_cache from 5400 to 3600 (attempt to alleviate memory swapping problems)
table_cache = 3600
thread_cache_size = 384
wait_timeout = 90
connect_timeout = 10
#080220 -- increased tmp_table_size and max_heap_table_size from 384M because created_tmp_disk_tables = 203k
#080319 -- decreased tmp_table_size and max_heap_table_size from 1024M to 768 (attempt to alleviate memory swapping problems)
tmp_table_size = 768M
max_heap_table_size = 768M
max_allowed_packet = 64M
max_connect_errors = 10
bulk_insert_buffer_size = 8M
query_cache_limit = 8M
query_cache_size = 512M
query_cache_type = 1
query_prealloc_size = 163840
query_alloc_block_size = 32768

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = -10
open_files_limit = 8192

[mysqldump]
quick
max_allowed_packet = 16M

[myisamchk]
key_buffer = 384M
sort_buffer = 384M
read_buffer = 256M
write_buffer = 256M

[mysqlhotcopy]
interactive-timeout


apache.conf
With edits to protect identity.

Code:
#
# Based upon the NCSA server configuration files originally by Rob McCool.
#
# This is the main Apache server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See <URL:http://httpd.apache.org/docs-2.1/> for detailed information about
# the directives.
#
# Do NOT simply read the instructions in here without understanding
# what they do.  They're here only as hints or reminders.  If you are unsure
# consult the online docs. You have been warned.
#
# The configuration directives are grouped into three basic sections:
#  1. Directives that control the operation of the Apache server process as a
#     whole (the 'global environment').
#  2. Directives that define the parameters of the 'main' or 'default' server,
#     which responds to requests that aren't handled by a virtual host.
#     These directives also provide default values for the settings
#     of all virtual hosts.
#  3. Settings for virtual hosts, which allow Web requests to be sent to
#     different IP addresses or hostnames and have them handled by the
#     same Apache server process.
#
# Configuration and logfile names: If the filenames you specify for many
# of the server's control files begin with "/" (or "drive:/" for Win32), the
# server will use that explicit path.  If the filenames do *not* begin
# with "/", the value of ServerRoot is prepended -- so "/var/log/apache2/foo.log"
# with ServerRoot set to "" will be interpreted by the
# server as "//var/log/apache2/foo.log".
#

### Section 1: Global Environment
#
# The directives in this section affect the overall operation of Apache,
# such as the number of concurrent requests it can handle or where it
# can find its configuration files.
#

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE!  If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the LockFile documentation (available
# at <URL:http://httpd.apache.org/docs-2.1/mod/mpm_common.html#lockfile>);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
ServerRoot "/etc/apache2"

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
#<IfModule !mpm_winnt.c>
#<IfModule !mpm_netware.c>
LockFile /var/lock/apache2/accept.lock
#</IfModule>
#</IfModule>

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
#
PidFile /var/run/apache2.pid

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive Off

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 15

##
## Server-Pool Size Regulation (MPM specific)
##

# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxClients: maximum number of server processes allowed to start
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_prefork_module>
    StartServers         50
    MinSpareServers      10
    MaxSpareServers      10
    ServerLimit          2000
    MaxClients           2000
    MaxRequestsPerChild  2000
</IfModule>

# worker MPM
# StartServers: initial number of server processes to start
# MaxClients: maximum number of simultaneous client connections
# MinSpareThreads: minimum number of worker threads which are kept spare
# MaxSpareThreads: maximum number of worker threads which are kept spare
# ThreadsPerChild: constant number of worker threads in each server process
# MaxRequestsPerChild: maximum number of requests a server process serves
<IfModule mpm_worker_module>
    ServerLimit           900
    StartServers          10
    MaxClients            10000
    MinSpareThreads       25
    MaxSpareThreads       75
    ThreadsPerChild       32
    MaxRequestsPerChild   100
</IfModule>

User www-data
Group www-data

#
# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride
# directive.
#

AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<Files ~ "^\.ht">
    Order allow,deny
    Deny from all
</Files>

TypesConfig /etc/mime.types

#
# DefaultType is the default MIME type the server will use for a document
# if it cannot otherwise determine one, such as from filename extensions.
# If your server contains mostly text or HTML documents, "text/plain" is
# a good value.  If most of your content is binary, such as applications
# or images, you may want to use "application/octet-stream" instead to
# keep browsers from trying to display binary files as though they are
# text.
#
DefaultType text/plain


#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog /var/log/apache2/error.log

#
# LogLevel: Control the number of messages logged to the error_log.
# Possible values include: debug, info, notice, warn, error, crit,
# alert, emerg.
#
LogLevel warn

# Include module configuration:
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf

# Include all the user configurations:
Include /etc/apache2/httpd.conf

# Include ports listing
Include /etc/apache2/ports.conf

# Include generic snippets of statements
Include /etc/apache2/conf.d/

#
# The following directives define some format nicknames for use with
# a CustomLog directive (see below).
#
# SITENAME adjusted common and combined log format for virtual hosting
#LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
#LogFormat "%h %l %u %t \"%r\" %>s %b" common
LogFormat "%v %h %l %u %t \"%r\" %>s %b" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

#
# ServerTokens
# This directive configures what you return as the Server HTTP response
# Header. The default is 'Full' which sends information about the OS-Type
# and compiled in modules.
# Set to one of:  Full | OS | Minor | Minimal | Major | Prod
# where Full conveys the most information, and Prod the least.
#
ServerTokens Prod

#
# Optionally add a line containing the server version and virtual host
# name to server-generated pages (internal error documents, FTP directory
# listings, mod_status and mod_info output etc., but not CGI generated
# documents or custom error documents).
# Set to "EMail" to also include a mailto: link to the ServerAdmin.
# Set to one of:  On | Off | EMail
#
ServerSignature Off

<IfModule alias_module>
    #
    # Aliases: Add here as many aliases as you need (with no limit). The format is
    # Alias fakename realname
    #
    # Note that if you include a trailing / on fakename then the server will
    # require it to be present in the URL.  So "/icons" isn't aliased in this
    # example, only "/icons/".  If the fakename is slash-terminated, then the
    # realname must also be slash terminated, and if the fakename omits the
    # trailing slash, the realname must also omit it.
    #
    # We include the /icons/ alias for FancyIndexed directory listings.  If
    # you do not use FancyIndexing, you may comment this out.
    #
    Alias /icons/ "/usr/share/apache2/icons/"

    <Directory "/usr/share/apache2/icons">
        Options Indexes MultiViews
        AllowOverride None
        Order allow,deny
        Allow from all
    </Directory>

</IfModule>

#
# Directives controlling the display of server-generated directory listings.
#
<IfModule mod_autoindex.c>

    #
    # IndexOptions: Controls the appearance of server-generated directory
    # listings.
    #
    IndexOptions FancyIndexing VersionSort HTMLTable NameWidth=*

    #
    # AddIcon* directives tell the server which icon to show for different
    # files or filename extensions.  These are only displayed for
    # FancyIndexed directories.
    #
    AddIconByEncoding (CMP,/icons/compressed.gif) x-compress x-gzip

    AddIconByType (TXT,/icons/text.gif) text/*
    AddIconByType (IMG,/icons/image2.gif) image/*
    AddIconByType (SND,/icons/sound2.gif) audio/*
    AddIconByType (VID,/icons/movie.gif) video/*

    AddIcon /icons/binary.gif .bin .exe
    AddIcon /icons/binhex.gif .hqx
    AddIcon /icons/tar.gif .tar
    AddIcon /icons/world2.gif .wrl .wrl.gz .vrml .vrm .iv
    AddIcon /icons/compressed.gif .Z .z .tgz .gz .zip
    AddIcon /icons/a.gif .ps .ai .eps
    AddIcon /icons/layout.gif .html .shtml .htm .pdf
    AddIcon /icons/text.gif .txt
    AddIcon /icons/c.gif .c
    AddIcon /icons/p.gif .pl .py
    AddIcon /icons/f.gif .for
    AddIcon /icons/dvi.gif .dvi
    AddIcon /icons/uuencoded.gif .uu
    AddIcon /icons/script.gif .conf .sh .shar .csh .ksh .tcl
    AddIcon /icons/tex.gif .tex
    AddIcon /icons/bomb.gif core

    AddIcon /icons/back.gif ..
    AddIcon /icons/hand.right.gif README
    AddIcon /icons/folder.gif ^^DIRECTORY^^
    AddIcon /icons/blank.gif ^^BLANKICON^^

    #
    # DefaultIcon is which icon to show for files which do not have an icon
    # explicitly set.
    #
    DefaultIcon /icons/unknown.gif

    #
    # AddDescription allows you to place a short description after a file in
    # server-generated indexes.  These are only displayed for FancyIndexed
    # directories.
    # Format: AddDescription "description" filename
    #
    #AddDescription "GZIP compressed document" .gz
    #AddDescription "tar archive" .tar
    #AddDescription "GZIP compressed tar archive" .tgz

    #
    # ReadmeName is the name of the README file the server will look for by
    # default, and append to directory listings.
    #
    # HeaderName is the name of a file which should be prepended to
    # directory indexes.
    ReadmeName README.html
    HeaderName HEADER.html

    #
    # IndexIgnore is a set of filenames which directory indexing should ignore
    # and not include in the listing.  Shell-style wildcarding is permitted.
    #
    IndexIgnore .??* *~ *# RCS CVS *,v *,t
</IfModule>

<IfModule mod_mime.c>

    #
    # AddType allows you to add to or override the MIME configuration
    # file mime.types for specific file types.
    #
    #AddType application/x-gzip .tgz
    #
    # AddEncoding allows you to have certain browsers uncompress
    # information on the fly. Note: Not all browsers support this.
    # Despite the name similarity, the following Add* directives have
    # nothing to do with the FancyIndexing customization directives above.
    #
    #AddEncoding x-compress .Z
    #AddEncoding x-gzip .gz .tgz
    #
    # If the AddEncoding directives above are commented-out, then you
    # probably should define those extensions to indicate media types:
    #
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz

    #
    # DefaultLanguage and AddLanguage allows you to specify the language of
    # a document. You can then use content negotiation to give a browser a
    # file in a language the user can understand.
    #
    # Specify a default language. This means that all data
    # going out without a specific language tag (see below) will
    # be marked with this one. You probably do NOT want to set
    # this unless you are sure it is correct for all cases.
    #
    # * It is generally better to not mark a page as
    # * being a certain language than marking it with the wrong
    # * language!
    #
    # DefaultLanguage nl
    #
    # Note 1: The suffix does not have to be the same as the language
    # keyword --- those with documents in Polish (whose net-standard
    # language code is pl) may wish to use "AddLanguage pl .po" to
    # avoid the ambiguity with the common suffix for perl scripts.
    #
    # Note 2: The example entries below illustrate that in some cases
    # the two character 'Language' abbreviation is not identical to
    # the two character 'Country' code for its country,
    # E.g. 'Danmark/dk' versus 'Danish/da'.
    #
    # Note 3: In the case of 'ltz' we violate the RFC by using a three char
    # specifier. There is 'work in progress' to fix this and get
    # the reference data for rfc1766 cleaned up.
    #
    # Catalan (ca) - Croatian (hr) - Czech (cs) - Danish (da) - Dutch (nl)
    # English (en) - Esperanto (eo) - Estonian (et) - French (fr) - German (de)
    # Greek-Modern (el) - Hebrew (he) - Italian (it) - Japanese (ja)
    # Korean (ko) - Luxembourgeois* (ltz) - Norwegian Nynorsk (nn)
    # Norwegian (no) - Polish (pl) - Portugese (pt)
    # Brazilian Portuguese (pt-BR) - Russian (ru) - Swedish (sv)
    # Simplified Chinese (zh-CN) - Spanish (es) - Traditional Chinese (zh-TW)
    #
    AddLanguage ca .ca
    AddLanguage cs .cz .cs
    AddLanguage da .dk
    AddLanguage de .de
    AddLanguage el .el
    AddLanguage en .en
    AddLanguage eo .eo
    AddLanguage es .es
    AddLanguage et .et
    AddLanguage fr .fr
    AddLanguage he .he
    AddLanguage hr .hr
    AddLanguage it .it
    AddLanguage ja .ja
    AddLanguage ko .ko
    AddLanguage ltz .ltz
    AddLanguage nl .nl
    AddLanguage nn .nn
    AddLanguage no .no
    AddLanguage pl .po
    AddLanguage pt .pt
    AddLanguage pt-BR .pt-br
    AddLanguage ru .ru
    AddLanguage sv .sv
    AddLanguage zh-CN .zh-cn
    AddLanguage zh-TW .zh-tw
</IfModule>

<IfModule mod_negotiation.c>
    #
    # LanguagePriority allows you to give precedence to some languages
    # in case of a tie during content negotiation.
    #
    # Just list the languages in decreasing order of preference. We have
    # more or less alphabetized them here. You probably want to change this.
    #
    LanguagePriority en ca cs da de el eo es et fr he hr it ja ko ltz nl nn no pl pt pt-BR ru sv zh-CN zh-TW

    #
    # ForceLanguagePriority allows you to serve a result page rather than
    # MULTIPLE CHOICES (Prefer) [in case of a tie] or NOT ACCEPTABLE (Fallback)
    # [in case no accepted languages matched the available variants]
    #
    ForceLanguagePriority Prefer Fallback

</IfModule>

<IfModule mod_mime.c>
    #
    # Specify a default charset for all pages sent out. This is
    # always a good idea and opens the door for future internationalisation
    # of your web site, should you ever want it. Specifying it as
    # a default does little harm; as the standard dictates that a page
    # is in iso-8859-1 (latin1) unless specified otherwise i.e. you
    # are merely stating the obvious. There are also some security
    # reasons in browsers, related to javascript and URL parsing
    # which encourage you to always set a default char set.
    #
    #AddDefaultCharset ISO-8859-1

    #
    # Commonly used filename extensions to character sets. You probably
    # want to avoid clashes with the language extensions, unless you
    # are good at carefully testing your setup after each change.
    # See http://www.iana.org/assignments/character-sets for the
    # official list of charset names and their respective RFCs.
    #
    AddCharset us-ascii    .ascii .us-ascii
    AddCharset ISO-8859-1  .iso8859-1  .latin1
    AddCharset ISO-8859-2  .iso8859-2  .latin2 .cen
    AddCharset ISO-8859-3  .iso8859-3  .latin3
    AddCharset ISO-8859-4  .iso8859-4  .latin4
    AddCharset ISO-8859-5  .iso8859-5  .cyr .iso-ru
    AddCharset ISO-8859-6  .iso8859-6  .arb .arabic
    AddCharset ISO-8859-7  .iso8859-7  .grk .greek
    AddCharset ISO-8859-8  .iso8859-8  .heb .hebrew
    AddCharset ISO-8859-9  .iso8859-9  .latin5 .trk
    AddCharset ISO-8859-10  .iso8859-10  .latin6
    AddCharset ISO-8859-13  .iso8859-13
    AddCharset ISO-8859-14  .iso8859-14  .latin8
    AddCharset ISO-8859-15  .iso8859-15  .latin9
    AddCharset ISO-8859-16  .iso8859-16  .latin10
    AddCharset ISO-2022-JP .iso2022-jp .jis
    AddCharset ISO-2022-KR .iso2022-kr .kis
    AddCharset ISO-2022-CN .iso2022-cn .cis
    AddCharset Big5        .Big5       .big5 .b5
    AddCharset cn-Big5     .cn-big5
    # For russian, more than one charset is used (depends on client, mostly):
    AddCharset WINDOWS-1251 .cp-1251   .win-1251
    AddCharset CP866       .cp866
    AddCharset KOI8      .koi8
    AddCharset KOI8-E      .koi8-e
    AddCharset KOI8-r      .koi8-r .koi8-ru
    AddCharset KOI8-U      .koi8-u
    AddCharset KOI8-ru     .koi8-uk .ua
    AddCharset ISO-10646-UCS-2 .ucs2
    AddCharset ISO-10646-UCS-4 .ucs4
    AddCharset UTF-7       .utf7
    AddCharset UTF-8       .utf8
    AddCharset UTF-16      .utf16
    AddCharset UTF-16BE    .utf16be
    AddCharset UTF-16LE    .utf16le
    AddCharset UTF-32      .utf32
    AddCharset UTF-32BE    .utf32be
    AddCharset UTF-32LE    .utf32le
    AddCharset euc-cn      .euc-cn
    AddCharset euc-gb      .euc-gb
    AddCharset euc-jp      .euc-jp
    AddCharset euc-kr      .euc-kr
    #Not sure how euc-tw got in - IANA doesn't list it???
    AddCharset EUC-TW      .euc-tw
    AddCharset gb2312      .gb2312 .gb
    AddCharset iso-10646-ucs-2 .ucs-2 .iso-10646-ucs-2
    AddCharset iso-10646-ucs-4 .ucs-4 .iso-10646-ucs-4
    AddCharset shift_jis   .shift_jis .sjis

    #
    # AddHandler allows you to map certain file extensions to "handlers":
    # actions unrelated to filetype. These can be either built into the server
    # or added with the Action directive (see below)
    #
    # To use CGI scripts outside of ScriptAliased directories:
    # (You will also need to add "ExecCGI" to the "Options" directive.)
    #
    #AddHandler cgi-script .cgi

    #
    # For files that include their own HTTP headers:
    #
    #AddHandler send-as-is asis

    #
    # For server-parsed imagemap files:
    #
    #AddHandler imap-file map

    #
    # For type maps (negotiated resources):
    # (This is enabled by default to allow the Apache "It Worked" page
    #  to be distributed in multiple languages.)
    #
    AddHandler type-map var

    #
    # Filters allow you to process content before it is sent to the client.
    #
    # To parse .shtml files for server-side includes (SSI):
    # (You will also need to add "Includes" to the "Options" directive.)
    #
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>

#
# Action lets you define media types that will execute a script whenever
# a matching file is called. This eliminates the need for repeated URL
# pathnames for oft-used CGI file processors.
# Format: Action media/type /cgi-script/location
# Format: Action handler-name /cgi-script/location
#

#
# Customizable error responses come in three flavors:
# 1) plain text 2) local redirects 3) external redirects
#
# Some examples:
#ErrorDocument 500 "The server made a boo boo."
#ErrorDocument 404 /missing.html
#ErrorDocument 404 "/cgi-bin/missing_handler.pl"
#ErrorDocument 402 http://www.example.com/subscription_info.html
#

#
# Putting this all together, we can internationalize error responses.
#
# We use Alias to redirect any /error/HTTP_<error>.html.var response to
# our collection of by-error message multi-language collections.  We use
# includes to substitute the appropriate text.
#
# You can modify the messages' appearance without changing any of the
# default HTTP_<error>.html.var files by adding the line:
#
#   Alias /error/include/ "/your/include/path/"
#
# which allows you to create your own set of files by starting with the
# /usr/share/apache2/error/include/ files and copying them to /your/include/path/,
# even on a per-VirtualHost basis.  The default include files will display
# your Apache version number and your ServerAdmin email address regardless
# of the setting of ServerSignature.
#
# The internationalized error documents require mod_alias, mod_include
# and mod_negotiation.  To activate them, uncomment the following 30 lines.

#    Alias /error/ "/usr/share/apache2/error/"
#
#    <Directory "/usr/share/apache2/error">
#        AllowOverride None
#        Options IncludesNoExec
#        AddOutputFilter Includes html
#        AddHandler type-map var
#        Order allow,deny
#        Allow from all
#        LanguagePriority en cs de es fr it nl sv pt-br ro
#        ForceLanguagePriority Prefer Fallback
#    </Directory>
#
#    ErrorDocument 400 /error/HTTP_BAD_REQUEST.html.var
#    ErrorDocument 401 /error/HTTP_UNAUTHORIZED.html.var
#    ErrorDocument 403 /error/HTTP_FORBIDDEN.html.var
#    ErrorDocument 404 /error/HTTP_NOT_FOUND.html.var
#    ErrorDocument 405 /error/HTTP_METHOD_NOT_ALLOWED.html.var
#    ErrorDocument 408 /error/HTTP_REQUEST_TIME_OUT.html.var
#    ErrorDocument 410 /error/HTTP_GONE.html.var
#    ErrorDocument 411 /error/HTTP_LENGTH_REQUIRED.html.var
#    ErrorDocument 412 /error/HTTP_PRECONDITION_FAILED.html.var
#    ErrorDocument 413 /error/HTTP_REQUEST_ENTITY_TOO_LARGE.html.var
#    ErrorDocument 414 /error/HTTP_REQUEST_URI_TOO_LARGE.html.var
#    ErrorDocument 415 /error/HTTP_UNSUPPORTED_MEDIA_TYPE.html.var
#    ErrorDocument 500 /error/HTTP_INTERNAL_SERVER_ERROR.html.var
#    ErrorDocument 501 /error/HTTP_NOT_IMPLEMENTED.html.var
#    ErrorDocument 502 /error/HTTP_BAD_GATEWAY.html.var
#    ErrorDocument 503 /error/HTTP_SERVICE_UNAVAILABLE.html.var
#    ErrorDocument 506 /error/HTTP_VARIANT_ALSO_VARIES.html.var

<IfModule mod_setenvif.c>
    #
    # The following directives modify normal HTTP response behavior to
    # handle known problems with browser implementations.
    #
    BrowserMatch "Mozilla/2" nokeepalive
    BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0
    BrowserMatch "RealPlayer 4\.0" force-response-1.0
    BrowserMatch "Java/1\.0" force-response-1.0
    BrowserMatch "JDK/1\.0" force-response-1.0

    #
    # The following directive disables redirects on non-GET requests for
    # a directory that does not include the trailing slash.  This fixes a
    # problem with Microsoft WebFolders which does not appropriately handle
    # redirects for folders with DAV methods.
    # Same deal with Apple's DAV filesystem and Gnome VFS support for DAV.
    #
    BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully
    BrowserMatch "MS FrontPage" redirect-carefully
    BrowserMatch "^WebDrive" redirect-carefully
    BrowserMatch "^WebDAVFS/1.[012]" redirect-carefully
    BrowserMatch "^gnome-vfs/1.0" redirect-carefully
    BrowserMatch "^XML Spy" redirect-carefully
    BrowserMatch "^Dreamweaver-WebDAV-SCM1" redirect-carefully
</IfModule>

#<IfModule mod_status.c>
    #
    # Allow server status reports generated by mod_status,
    # with the URL of http://servername/server-status
    # Change the ".example.com" to match your domain to enable.
    #
    #<Location /server-status>
    #    SetHandler server-status
    #    Order deny,allow
    #    Deny from all
    #    Allow from .example.com
    #</Location>
#</IfModule>

#<IfModule mod_info.c>
    #
    # Allow remote server configuration reports, with the URL of
    #  http://servername/server-info (requires that mod_info.c be loaded).
    # Change the ".example.com" to match your domain to enable.
    #
    #<Location /server-info>
    #    SetHandler server-info
    #    Order deny,allow
    #    Deny from all
    #    Allow from .example.com
    #</Location>
#</IfModule>

# SITENAME
# BufferLogs is experimental http://httpd.apache.org/docs/2.0/mod/mod_log_config.html
BufferedLogs On
# htpasswd info stored in national database
Auth_MySQL_Info localhost username password

# Include the virtual host configurations:
Include /etc/apache2/sites-enabled/


typical vhost config
With edits to protect identity.

Code:
<VirtualHost 111.222.333.444:80>
ServerName sitename.com
ServerAlias www.sitename.com
RewriteEngine On

# force www on URL
RewriteCond %{HTTP_HOST} !^www.sitename.com$ [NC]
RewriteRule ^(.*)$ http://www.sitename.com$1 [R,L]

# php mail return-path
php_admin_value sendmail_path "/usr/sbin/sendmail -t -i -f bounce@sitename.org"

DocumentRoot /home/sitename/public_html
ScriptAlias /cgi-bin/ "/home/sitename/cgi-bin/"

# public directory restrictions
<Directory /home/sitename/public_html>
Options -Indexes IncludesNOEXEC FollowSymLinks
allow from all
</Directory>

# siteadmin control panel authentication
<Directory ~ "^/home/.*/public_html/.*s/admin">
AuthName "SA Control Panel"
AuthType Basic
AuthUserFile /dev/null
AuthBasicAuthoritative Off
AuthMYSQL On
AuthMySQL_Authoritative On
AuthMySQL_DB national
AuthMySQL_Password_Table htpasswd
AuthMySQL_Group_Table htpasswd
AuthMySQL_Username_Field username
AuthMySQL_Password_Field passwd
AuthMySQL_Group_Field groups
AuthMySQL_Encryption_Types Plaintext Crypt_DES
AuthMySQL_Empty_Passwords off
# impose access restrictions on php, html, cgi or / requests only
<FilesMatch "(?:php|html?|cgi|\/$)">
   require group sitename manager admin
</FilesMatch>
</Directory>

</VirtualHost>


3) Do you have access logs enabled on apache?
No, we have recently disabled access logging to reduce IO. We now log only errors.

4) What's the language used mostly on the backend? If it's PHP, do you have mod_php or CGI/FastCGI?
PHP is the primary language on > 95% of the site. We use mod_php and I believe just standard CGI (not FastCGI).

XCache
Figured you might want this too. XCache has been very helpful.

Code:
[xcache.admin]
;SITENAME this is off because admin in /manage directory
xcache.admin.enable_auth = Off
xcache.admin.user = "user"
; xcache.admin.pass = md5($your_password)
xcache.admin.pass = "md5hashliveshere"

[xcache]
; ini only settings, all the values here is default unless explained

; select low level shm/allocator scheme implemenation
xcache.shm_scheme =        "mmap"
; to disable: xcache.size=0
; to enable : xcache.size=64M etc (any size > 0) and your system mmap allows
xcache.size  =                64M
; set to cpu count (cat /proc/cpuinfo |grep -c processor)
xcache.count =                2
; just a hash hints, you can always store count(items) > slots
xcache.slots =                8K
; ttl of the cache item, 0=forever
xcache.ttl   =                 7200
; interval of gc scanning expired items, 0=no scan, other values is in seconds
xcache.gc_interval =           300

; same as aboves but for variable cache
xcache.var_size  =            16M
xcache.var_count =             2
xcache.var_slots =            8K
; default ttl
xcache.var_ttl   =             7200
xcache.var_maxttl   =          14400
xcache.var_gc_interval =     300

xcache.test =                Off
; N/A for /dev/zero
xcache.readonly_protection = Off
; for *nix, xcache.mmap_path is a file path, not directory.
; Use something like "/tmp/xcache" if you want to turn on ReadonlyProtection
; 2 group of php won't share the same /tmp/xcache
; for win32, xcache.mmap_path=anonymous map name, not file path
xcache.mmap_path =    "/dev/zero"

; leave it blank(disabled) or "/tmp/phpcore/"
; make sure it's writable by php (without checking open_basedir)
xcache.coredump_directory =   ""

; per request settings
xcache.cacher =               On
xcache.stat   =               On
xcache.optimizer =           Off

[xcache.coverager]
; per request settings
; enable coverage data collecting for xcache.coveragedump_directory and xcache_coverager_start/stop/get/clean() functions (will hurt executing performance)
xcache.coverager =          Off

; ini only settings
; make sure it's readable (care open_basedir) by coverage viewer script
; requires xcache.coverager=On
xcache.coveragedump_directory = ""


Thanks in advance for sharing your wisdom!

06-12-2008 05:52 PM
Find all posts by this user Quote this message in a reply
kevin
Junior Member
**

Posts: 3
Group: Registered
Joined: Jun 2008
Status: Offline
Reputation: 0
Post: #4
RE: Nonprofit requesting ideas for busy server

Here's an example of system drag. You can see that the swap is being used now (biggest offender?) There are about 850 simultaneous users now, but I've seen "snappy" all the way up to about 900. There are probably high-memory queries running right now and it has pushed the system over the edge.

free -m

Code:
.............total       used       free     shared    buffers     cached
Mem:          3995       3024        971          0         28        824
-/+ buffers/cache:       2171       1824
Swap:         9318       1220       8097


top

Code:
top - 19:31:32 up 137 days,  6:13,  2 users,  load average: 20.35, 36.12, 43.15
Tasks: 156 total,   4 running, 141 sleeping,   0 stopped,  11 zombie
Cpu(s): 61.2%us, 13.4%sy,  0.0%ni, 15.4%id,  8.7%wa,  0.3%hi,  1.0%si,  0.0%st
Mem:   4091784k total,  2946196k used,  1145588k free,    29208k buffers
Swap:  9542568k total,  1250232k used,  8292336k free,   840136k cached

06-12-2008 07:38 PM
Find all posts by this user Quote this message in a reply
Pacifier
Administrator
*******

Posts: 214
Group: Administrators
Joined: May 2006
Status: Offline
Reputation: 0
Post: #5
RE: Nonprofit requesting ideas for busy server

I was expecting some more details in the top output. Can you please let me know which processes use most of the CPU / memory in top, i.e. which are usually located upwards. Since you said heavy queries are to blame, let's assume it's the MySQL that's at fault.

Anyway, do let me know the output of following commands:

ps aux | grep -c httpd
ps aux | grep -c mysqld

Try the following:

1) You mentioned that you use CGI for PHP at some places. It will be better to use mod_php in all instances.

2) For the MySQL part, there are few things that you can try:
* reduce the key_buffer_size if it's not being used fully. Check your statistics from PhpMyAdmin -> MySQL runtime information in key_reads and key_writes. Try to provide me that output as well.
* disable query cache at all if your queries are random and fetch totally different content most of the time. It may help reduce some useless overhead.

I will be able to offer more recommendations for the MySQL part once you provide me that phpMyAdmin page; it's quite helpful.

3) Change the following in apache.conf/httpd.conf:

Code:
Timeout 60
KeepAlive On

MaxKeepAliveRequests 200
KeepAliveTimeout 4


I will be able to give you further assistance once you post the output of the commands that I mentioned above, and output of:
httpd -V (might be apache -V or apachectl -V)

Moreover, I will either need you to let me know the modules you have enabled (and why), or post the content of following two files:
Include /etc/apache2/mods-enabled/*.load
Include /etc/apache2/mods-enabled/*.conf

06-15-2008 09:23 AM

This post was last modified: 06-15-2008 09:24 AM by Pacifier.

Find all posts by this user Quote this message in a reply
Post Reply  Post Thread 

View a Printable Version
Send this Thread to a Friend
Subscribe to this Thread | Add Thread to Favorites

Forum Jump: