i simply have no clue. it/s night, server gets very few hits, yet the load stays like this.

during day it goes up to 70 80 .

this is a p4 3.0 with 2 gb ram,

centos 4.4 mysql 5.0.27 php 4.4.4

top - 02:46:54 up 1 day, 9:21, 2 users, load average: 1.80, 1.72, 1.59
Tasks: 151 total, 1 running, 150 sleeping, 0 stopped, 0 zombie
Cpu(s): 2.6% us, 1.5% sy, 0.0% ni, 89.8% id, 2.0% wa, 0.0% hi, 4.1% si
Mem: 2000552k total, 1928832k used, 71720k free, 117540k buffers
Swap: 4192956k total, 1964k used, 4190992k free, 1330272k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
15028 lafamil 15 0 17596 9380 1840 S 1 0.5 0:00.03 cpsrvd-ssl
19566 root 15 0 17588 9076 1636 S 1 0.5 0:01.52 cpsrvd-ssl
32448 named 24 0 46044 9.8m 1996 S 0 0.5 18:34.37 named
12217 radiomag 15 0 33324 3968 1200 S 0 0.2 11:35.77 sc_serv
12871 radiotin 15 0 29468 2484 912 S 0 0.1 2:44.98 sc_serv
27805 root 15 0 2100 1084 800 R 0 0.1 0:00.04 top
1 root 15 0 1752 592 508 S 0 0.0 0:02.38 init
2 root RT 0 0 0 0 S 0 0.0 0:00.25 migration/0
3 root 34 19 0 0 0 S 0 0.0 0:32.57 ksoftirqd/0
4 root RT 0 0 0 0 S 0 0.0 0:00.54 watchdog/0
5 root RT 0 0 0 0 S 0 0.0 0:00.01 migration/1
6 root 34 19 0 0 0 S 0 0.0 0:00.54 ksoftirqd/1
7 root RT 0 0 0 0 S 0 0.0 0:00.00 watchdog/1
8 root 10 -5 0 0 0 S 0 0.0 0:14.10 events/0
9 root 10 -5 0 0 0 S 0 0.0 0:00.15 events/1
10 root 10 -5 0 0 0 S 0 0.0 0:01.29 khelper
11 root 10 -5 0 0 0 S 0 0.0 0:00.00 kthread
15 root 10 -5 0 0 0 S 0 0.0 0:01.81 kblockd/0
16 root 10 -5 0 0 0 S 0 0.0 0:00.78 kblockd/1
17 root 17 -5 0 0 0 S 0 0.0 0:00.00 kacpid
103 root 10 -5 0 0 0 S 0 0.0 0:00.00 kseriod
163 root 10 -5 0 0 0 S 0 0.0 6:57.48 kswapd0
164 root 16 -5 0 0 0 S 0 0.0 0:00.00 aio/0
165 root 17 -5 0 0 0 S 0 0.0 0:00.00 aio/1
332 root 12 -5 0 0 0 S 0 0.0 0:00.00 kpsmoused
23892 root 14 -5 0 0 0 S 0 0.0 0:00.00 ata/0
19711 root 14 -5 0 0 0 S 0 0.0 0:00.00 ata/1
13794 root 14 -5 0 0 0 S 0 0.0 0:00.00 ata_aux
30504 root 13 -5 0 0 0 S 0 0.0 0:00.00 scsi_eh_0
764 root 11 -5 0 0 0 S 0 0.0 0:00.00 scsi_eh_1
4997 root 10 -5 0 0 0 S 0 0.0 1:29.60 kjournald
14450 root 10 -5 1632 492 416 S 0 0.0 0:00.01 udevd
13833 root 11 -5 0 0 0 S 0 0.0 0:00.00 kauditd
18272 root 12 -5 0 0 0 S 0 0.0 0:00.00 kmirrord
24824 root 10 -5 0 0 0 S 0 0.0 0:05.94 kjournald
8554 root 10 -5 0 0 0 S 0 0.0 0:02.60 kjournald
18492 root 18 0 1640 556 460 S 0 0.0 0:11.57 syslogd
29308 root 15 0 1584 384 316 S 0 0.0 0:00.61 klogd
21726 root 18 0 1592 300 220 S 0 0.0 0:02.60 irqbalance
1480 root 18 0 1576 400 332 S 0 0.0 0:00.00 courierlogger
8904 root 18 0 1892 628 516 S 0 0.0 0:00.00 authdaemond
13883 root 18 0 1892 376 252 S 0 0.0 0:00.30 authdaemond
31527 root 18 0 1892 376 252 S 0 0.0 0:00.30 authdaemond
2667 root 18 0 1892 376 252 S 0 0.0 0:00.34 authdaemond
26391 root 15 0 1892 376 252 S 0 0.0 0:00.33 authdaemond
15979 root 15 0 1892 376 252 S 0 0.0 0:00.36 authdaemond
16310 root 16 0 1876 512 300 S 0 0.0 0:00.06 smartd
12974 root 24 0 1552 500 424 S 0 0.0 0:00.00 acpid
25637 root 18 0 4132 1036 744 S 0 0.1 0:00.05 sshd
3530 root 15 0 2152 764 636 S 0 0.0 0:00.00 xinetd
17525 root 18 0 1552 400 332 S 0 0.0 0:00.27 courierlogger

root@nemesis [~]# free -m
total used free shared buffers cached
Mem: 1953 1897 56 0 120 1304
-/+ buffers/cache: 471 1481
Swap: 4094 1 4092

Which version of kernel are you using? Post output of (you can mask your hostname with xx)

top - 19:08:58 up 2 days, 1:43, 5 users, load average: 42.73, 40.79, 28.12
Tasks: 432 total, 22 running, 407 sleeping, 1 stopped, 2 zombie
Cpu(s): 42.2% us, 17.1% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 40.7% si
Mem: 2000552k total, 1925864k used, 74688k free, 116020k buffers
Swap: 4192956k total, 1440k used, 4191516k free, 918572k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26714 mysql 25 0 107m 20m 4048 R 30 1.1 3:31.30 mysqld
610 radio6m 15 0 39548 10m 1092 S 24 0.5 19:07.78 sc_serv
21361 emuzica 18 0 13856 4376 3084 R 7 0.2 0:00.21 php
22191 emuzica 18 0 0 0 0 R 6 0.0 0:00.20 php
24106 emuzica 18 0 13624 3724 2688 R 6 0.2 0:00.20 php
29326 root 16 0 2260 1272 800 R 6 0.1 0:01.25 top
27149 nobody 16 0 10376 6716 1108 S 4 0.3 0:00.33 httpd
30634 emuzica 18 0 14276 5188 3432 R 4 0.3 0:00.13 php
27094 nobody 15 0 10376 6724 1112 S 4 0.3 0:00.50 httpd
15113 emuzica 18 0 13284 2196 1708 R 4 0.1 0:00.12 php
32518 lafamil 18 0 18140 9384 3800 R 4 0.5 0:00.35 php
8112 educativ 16 0 18368 9472 3716 S 4 0.5 0:00.29 php
7241 nobody 15 0 10376 6708 1100 S 3 0.3 0:00.29 httpd
19012 nobody 15 0 10376 6724 1112 S 3 0.3 0:00.90 httpd
15276 erdeeshl 17 0 14232 5068 3472 S 2 0.3 0:00.12 php
1581 nobody 15 0 10376 6720 1108 S 2 0.3 0:00.70 httpd
16731 educativ 16 0 15796 6936 3692 S 2 0.3 0:00.30 php
12217 radiomag 15 0 33324 3968 1200 S 1 0.2 14:00.00 sc_serv
17246 nobody 15 0 10376 6712 1108 S 1 0.3 0:00.82 httpd
5718 nobody 15 0 10376 6756 1144 S 1 0.3 0:00.74 httpd
27578 nobody 15 0 10760 6848 1236 S 1 0.3 0:00.78 httpd
7023 nobody 15 0 10376 6760 1148 S 1 0.3 0:00.71 httpd
2844 nobody 17 0 10376 6724 1112 S 1 0.3 0:00.59 httpd

root@nemesis [~/server_tweak]# uname -a
Linux nemesis.design-services.us 2.6.18.2-grs-grsec #4 SMP Sat Nov 18 22:58:27 CET 2006 i686 i686 i386 GNU/Linux

Timeout 10
KeepAlive On
MaxKeepAliveRequests 150
KeepAliveTimeout 4
MinSpareServers 10
MaxSpareServers 20
StartServers 15
MaxClients 1000
MaxRequestsPerChild 0

Timeout 10
KeepAlive On
MaxKeepAliveRequests 150
KeepAliveTimeout 4
MinSpareServers 10
MaxSpareServers 20
StartServers 15
MaxClients 1000
MaxRequestsPerChild 0

root@nemesis [/usr/local/apache/conf]# uname -a
Linux nemesis.design-services.us 2.6.18.2-grs-grsec #4 SMP Sat Nov 18 22:58:27 CET 2006 i686 i686 i386 GNU/Linux

root@nemesis [/usr/local/apache/conf]# top
top - 19:10:27 up 2 days, 1:45, 5 users, load average: 33.81, 38.75, 28.53
Tasks: 196 total, 12 running, 182 sleeping, 1 stopped, 1 zombie
Cpu(s): 41.9% us, 22.7% sy, 0.0% ni, 1.0% id, 0.5% wa, 0.3% hi, 33.6% si
Mem: 2000552k total, 1667224k used, 333328k free, 115344k buffers
Swap: 4192956k total, 1420k used, 4191536k free, 920072k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
610 radio6m 15 0 39548 10m 1092 S 12 0.5 19:18.08 sc_serv
29972 emuzica 25 0 0 0 0 Z 7 0.0 0:00.22 php <defunct>
29037 phunro 23 0 15260 6500 3804 S 7 0.3 0:00.20 php
4859 postgres 20 0 7708 6292 3372 S 6 0.3 0:00.17 postgresadmin
10761 emuzica 25 0 14512 5656 3748 R 5 0.3 0:00.16 php
4830 phunro 25 0 15188 6204 3608 R 5 0.3 0:00.15 php
26714 mysql 15 0 107m 20m 4068 S 4 1.1 3:51.27 mysqld
11928 emuzica 18 0 13724 3780 2688 R 3 0.2 0:00.10 php
32511 listen 15 0 30460 2896 1004 S 3 0.1 5:56.24 sc_serv
11025 nobody 16 0 10428 6620 1084 S 3 0.3 0:00.08 httpd
25467 erdeeshl 16 0 31504 21m 9668 S 2 1.1 0:00.35 cpanel
12503 nobody 15 0 10376 6628 1040 S 2 0.3 0:00.06 httpd
9556 nobody 15 0 10376 6660 1088 S 2 0.3 0:00.06 httpd
12871 radiotin 15 0 29468 2484 912 S 2 0.1 3:32.65 sc_serv
14844 nobody 15 0 10376 6600 1036 S 2 0.3 0:00.05 httpd
28651 named 25 0 40924 4544 1968 S 1 0.2 0:07.64 named
145 nobody 15 0 10376 6600 1040 S 1 0.3 0:00.04 httpd
12217 radiomag 15 0 33324 3968 1200 S 1 0.2 14:00.87 sc_serv
25903 mailnull 15 0 8320 3604 2348 S 1 0.2 0:02.51 exim
21590 root 15 0 2268 1240 800 R 1 0.1 0:00.63 top
13718 nobody 15 0 10376 6648 1052 S 1 0.3 0:00.03 httpd
18221 nobody 22 0 10376 6620 1028 S 1 0.3 0:00.03 httpd
739 nobody 18 0 10376 6588 1024 S 1 0.3 0:00.03 httpd
14305 root 23 0 10376 7396 1860 R 1 0.4 1:13.94 httpd
18782 nobody 15 0 10376 6568 1032 S 1 0.3 0:00.02 httpd
26011 nobody 16 0 10376 6616 1044 S 1 0.3 0:00.02 httpd
31567 radiomed 15 0 33344 2952 1308 S 0 0.1 7:12.27 sc_serv

top - 19:18:01 up 2 days, 1:52, 5 users, load average: 74.91, 53.90, 37.94
Tasks: 421 total, 69 running, 350 sleeping, 1 stopped, 1 zombie
Cpu(s): 44.7% us, 19.9% sy, 0.0% ni, 0.5% id, 0.0% wa, 0.4% hi, 34.5% si
Mem: 2000552k total, 1820304k used, 180248k free, 117952k buffers
Swap: 4192956k total, 1420k used, 4191536k free, 907972k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
28270 mysql 15 0 104m 18m 4036 S 46 0.9 1:40.09 mysqld
8332 nobody 15 0 10376 6748 1144 S 10 0.3 0:00.54 httpd
31944 autosale 20 0 24812 15m 3580 R 7 0.8 0:02.02 php
610 radio6m 15 0 39548 10m 1080 S 6 0.6 20:18.30 sc_serv
9502 nobody 15 0 10376 6696 1092 S 6 0.3 0:00.31 httpd
7169 nobody 15 0 10376 6744 1136 S 6 0.3 0:01.12 httpd
20023 emuzica 17 0 14356 5208 3460 R 5 0.3 0:00.15 php
20864 nobody 15 0 10376 6720 1120 S 5 0.3 0:00.51 httpd
29162 mailnull 16 0 8436 3200 1828 S 5 0.2 0:00.14 exim
24303 root 20 0 7540 4036 1336 R 4 0.2 0:00.14 pureauth
24981 lafamil 17 0 14844 5616 3380 R 4 0.3 0:00.13 php
20411 educativ 17 0 14012 5000 3560 R 4 0.2 0:00.13 php
5737 educativ 17 0 15424 6468 3688 R 4 0.3 0:00.13 php
15730 root 16 0 2280 1240 800 R 4 0.1 0:00.24 top
17581 emuzica 18 0 13592 3704 2688 R 4 0.2 0:00.12 php
3706 emuzica 17 0 13596 3680 2688 R 4 0.2 0:00.12 php
31581 erdeeshl 17 0 13636 3728 2688 R 4 0.2 0:00.11 php
27636 emuzica 18 0 0 0 0 R 4 0.0 0:00.11 php
11982 root 20 0 0 0 0 R 3 0.0 0:00.22 whostmgr
9906 emuzica 25 0 13760 4048 2844 R 3 0.2 0:00.10 php
3126 emuzica 25 0 14480 5648 3744 S 3 0.3 0:00.14 php
25407 emuzica 22 0 13592 3776 2688 R 3 0.2 0:00.10 php
5940 root 19 0 17436 7632 3752 R 3 0.4 0:00.10 whostmgr
307 emuzica 18 0 14392 5224 3468 R 3 0.3 0:00.10 php
25508 emuzica 17 0 13868 4488 3160 R 3 0.2 0:00.10 php
20861 phunro 17 0 13840 4320 3044 R 3 0.2 0:00.10 php
1579 educativ 17 0 14012 4688 3320 R 3 0.2 0:00.10 php
24292 admin 18 0 13500 3640 2688 R 3 0.2 0:00.10 php
20882 emuzica 18 0 13636 3656 2688 R 3 0.2 0:00.10 php
19546 nobody 15 0 10376 6796 1168 S 3 0.3 0:00.63 httpd
6461 root 19 0 20552 18m 8912 S 3 1.0 0:00.21 whostmgr
23864 emuzica 18 0 13628 3676 2688 R 3 0.2 0:00.08 php
20204 emuzica 17 0 13612 3704 2688 R 2 0.2 0:00.07 php
6285 nobody 15 0 10376 6676 1068 S 2 0.3 0:00.19 httpd
27340 emuzica 17 0 13596 3732 2688 R 2 0.2 0:00.06 php
31792 educativ 17 0 13636 3684 2688 R 2 0.2 0:00.06 php
681 emuzica 17 0 13600 3660 2688 R 2 0.2 0:00.06 php
18759 nobody 15 0 10376 6708 1100 S 2 0.3 0:00.15 httpd
11471 phunro 18 0 15276 6484 3784 R 2 0.3 0:00.19 php
31567 radiomed 15 0 33344 2984 1304 S 1 0.1 7:17.91 sc_serv
12217 radiomag 15 0 33324 3968 1200 S 1 0.2 14:09.10 sc_serv
6652 nobody 15 0 10376 6732 1112 S 1 0.3 0:00.74 httpd
21265 root 19 0 17140 5600 2872 R 1 0.3 0:00.04 whostmgr
10133 neo 15 0 5812 4604 1004 S 1 0.2 4:19.82 ircd.2006090101
28651 named 25 0 40924 4772 1972 S 1 0.2 0:12.28 named
26011 nobody 15 0 10376 6724 1112 S 1 0.3 0:00.77 httpd
5989 nobody 15 0 10376 6756 1148 S 1 0.3 0:00.38 httpd
14070 nobody 15 0 10376 6720 1116 S 1 0.3 0:00.32 httpd

Maybe you're just reaching the capacity there. Try turning off the sc_serv (shoutcast server?) and ircd.xx and it should help somewhat.

Post output of ps aux | grep -c httpd and my.cnf .. I am sure you do need mysql optimizations as well. Also I would like to know what type of sites are hosted on the server?

root@nemesis [~]# ps aux | grep -c httpd
92

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
# Default to using old password format for compatibility with mysql 3.x
# clients (those using the mysqlclient10 compatibility package).
#old_passwords=1

[mysql.server]
user=mysql
# basedir=/var/lib
query_cache_limit=50M
query_cache_size=256M
query_cache_type=1

max_connections = 448
max_user_connections = 448

interactive_timeout=60
wait_timeout=60
connect_timeout=10

thread_cache_size=16
key_buffer=96M

max_allowed_packet=1M
table_cache=1024
record_buffer=1M
sort_buffer_size=2M
read_buffer_size=2M
read_rnd_buffer_size = 2M

thread_concurrency=2
myisam_sort_buffer_size=64M
server-id=1

[mysqld_safe]
err-log=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid


[mysqldump]
quick
max_allowed_packet = 16M

[mysql]
no-auto-rehash

[isamchk]
key_buffer=64M
sort_buffer=64M
read_buffer=2M
write_buffer=2M

[mysqlhotcopy]
interactive-timeout

there are around 115 websites on the server, using mysql database and postgresql database. the thing is that even at night when the server is not being used, the load never drops lower than 3 4

That is probably because of sc_servd... SHOUTCAST server...

You should probably lower your query_cache size and limit in my.cnf unless you have a lot of sites that do lots of reads but NO WRITES! Try lowering it to 1m and 16m.

Make the following changes in httpd.conf:

Timeout of 10 is way too low and will end up in more refreshes unless you have all very high latency visitors on good quality broadband.

Do a "ps aux" to see the list of running processes.

This post was last modified: 11-24-2006 11:06 AM by Pacifier.