AP@Linux Disk IO 效能調校

針對本次的測試說明如下:
1.使用Bryan提供的測試程式(XGW.STK.K1，以下簡稱XGW)(內含56000多筆的資料)來進行測試,
   壓力測試程式在Windows 7(小黑nb,172.16.43.144)上執行
   同時也執行了模擬後端主機(dbs)接收的的程式
2.本次的XGW在3台機器上分別進行驗證,不過主要以驗證PC -1為主,硬體與作業系統規格如下


PC - 1
kernel
2.6.18-164.el5PAE
distribution
RHEL5.4(172.16.43.99)
memory
MemTotal: 3916364 kB
cpu
Intel(R) Core(TM)2 Quad CPU Q8400 @ 2.66GHz
disk
500G 7200rpm(single)
PC - 2
kernel
2.6.18-92.el5
distribution
RHEL5.2(172.16.43.171)
memory
MemTotal: 3107452 kB
cpu
Intel(R) Pentium(R) Dual CPU E2140 @ 1.60GHz
disk
160G 7200rpm(single)
PC - 3
kernel
2.6.18-194.el5 
distribution
CentOS5.5(172.16.43.170)
memory
MemTotal: 16411680 kB
cpu
Intel(R) Xeon(R) CPU E5506 @ 2.13GHz(quard core * 2)
disk
7200 rpm(raid-5, 146G*3)

3.本次測試以上3台都同樣在172.16.43.0/24 網段上進行,所以應該不會有網路的堵塞問題
   曾經試過對172.16.53.20進行測試,都會莫名其妙增加很多時間,推測,可能是172.16.53.0/24上有人大量丟封包


測試方法說明:
1.先在PC - 1進行完整的測試
   (1)以作業系統剛安裝好的參數,先進行一次測試,當成參考值
   (2)針對Block Device的各項參數進行調整(依開發人員描述,主要為disk io所影響)
   (3)測試時,以相同的壓測步驟與XGW初始化環境一一進行驗證
   (4)測試期間收集送完56000筆的時間,XGW主機的整體cpu使用狀況,單一支XGW使用cpu狀況,系統wait io的狀況進行資料收集
2.從1.所測試出來的較佳幾個系統調整狀況,在PC-2及PC-3上面也進行驗證
   (1)在較老的機器上(PC-2)進行驗證
   (2)在較佳的機器上(PC-3, 有做raid-5)進行驗證
   (3)以上2個再跟PC-1進行對照驗證
3.在PC-1上安裝ramdisk同時驗證效能


測試結果如下:

黃色那一行是作業系統出廠的預設參數所測出來的結果(nr_requests 預設為128,預設為cfq)
土色那一行是調整後,得到較佳數據的2個結果, wa(wait for disk IO)的狀況有呈現較好的狀況,數據也反應在time欄位上
紫色是依據PC-1的結果,也同樣在PC-2 , PC-3上面驗證的結果


HDD
nr_requests
episode
time(sec)
min CPU idle%
avg CPU idle%
XGW useage%
wa%(inc. in idle)
PC2 time
PC3 time
cfq
64
0c0101092
173

128
0c0202092
157
44
68.09
45.66
19.2
168
82
512
0c01020c1
121

1024
0c01030c1
85

2048
0c0103092
86
115
66
4096
0c01010c1
93
35
63.26
56.06
18.65
116
65
anticipatory
64
0b0101012
159

128
0b0202012
153

2048
0b0103012
85
114
71
4096
0b01010c1
77
55
72.88
62.8
11.04
115
69
deadline
64
0a0101012
171

128
0a0202012
159

2048
0a0103012
101
121
121
noop
64
0d01010b1
163

128
0d02020b1
171

2048
0d01030b1
140
181
231




ramdisk
nr_requests
episode
time diff
cfq
128
-
79
1024
-
79
2048
-
76
anticipatory
128
-
79

結論與建議
1. 從結果可以得知
     (1)nr_requests(queue length)增加,對系統disk io的效能確實有效果(nr_requests有助減少seek time)
     (2)ramdisk因為沒有磁頭,沒有seek time的issue,然而效能本身己經很好
     (3)raid-5 (pc-3)的效能,在經過調整後,在本案的情境下,可以比ramdisk更好
2. 建議
     (1)依以下調整參數(建議先使用"cfq" , nr_requests=4096)
        在/etc/rc.local上面新增一筆
         echo 4096 > /sys/block/[disk device]/queue/nr_requests
         上面[disk device]請置換上XGW所安裝與執行的device上,例如sda(如果我記得沒錯,GW15跟16應該都是sda)
         重新開機後啟動XGW
         注意,nr_requests的增加,在系統異常時,愈有機會發生資料未完全寫入硬碟裡
                  異常發生時,建議確認XGW是否有漏寫資料,如果有發生,需要實際操作補送與寫入的作業(XGW跟後台間回報序號倒回重補的作業)
      (2)使用raid-5
         從 PC-3的結果可以看出raid-5的IO throughput本身就己經很優秀,再經調整後,
         已可以比PC-1的預設情境好上很多
      (3)使用ramdisk(非SSD)
         ramdisk本身是效能己經很好,不過因為他對nr_requests的設定是沒幫助的,不過效能穩定性高上許多
         再者,坊間對Ramdisk己可以支援raid,所以加乘使用,效果應該會更佳
    以上的建議,請事先驗證過後,再行開放,
    若有發生效率沒有明顯改善時,請確認壓測過程是否有網路的干擾因素(如前所述,可能會有網路堵塞或者效率不佳的因素)
補充說明:
  1. file fragmentation
    上次在客戶XGW上收集回來的資料,有對於XGW flush機制換版前後進行log檔結構分析
    在換版前(2011/03/17)的每個log(大檔,尤其是Ord,Rpt,RptOth及Sync log等),其fragmentation(檔案的破碎程度)
     大概都在100~300以上
     在換上新版後(2011/03/24)每個log(尤其是剛才那3個檔),其fragmentation都降到大概在10之間
     該次換版效能的改善從中可以發現一斑
     檔案的fragment會嚴重影響IO的效能,
  2. elevator
    (1)noop
       不使用任何elevator機制
     (2)deadline
       使用one-way的作業,在執行disk read或write時會去merge目前queue裡的 requests(有助降低seek time),對於"read"的作業更甚於"write",為了避免resource starvation,系統提供了許多的expire機制.這個elevator配合TQC及效率較佳的Disk,很適合拿來做資料庫系統
     (3)anticipatory
       在kernel 2.6版之前,這個elevator是預設的.在執行disk read或write時會去merge目前queue裡的 requests(有助降低seek time),read或write作業是分批作業進行.在進行資料讀取時,會多預讀一些資料起來,系統的預期是,可能接下來的動作是要再繼續往下讀一些,如此在某些情境下,可以減少磁頭的移動.或許針對某些資料庫可以提供很好的效率(在某些特定條件下)
       前面在的測試,在某些情況下,anticipatory反而比cfq好,或許跟XGW.STK.K1在這次的案例裡,幾乎只有write動作有關,所以系統做request merge的效率很好
     (4)cfq
       Complete Fair Queueing Scheduler, 這是最中庸的elevator,因為它是round robin的處理作業,針對各process提供相同的device IO requests數量.這個elevator在kernel 2.6之後,就是預設的.提供read and write的requests merging.這種elevator適用在多使用者的環境下(筆者另外解讀,在multi-process或multi-file-IO下也蠻適合的).
       本例在XGW.STK.K1是multi-thread會同時開啟許多個檔案進行write作業,所以cfq的作業在預計的nr_requests=128或許效果沒那麼好,可是在nr_requests愈來愈大的情況下,merging效果愈好,約在2048~4096就有很好的成效

     本次的測試,覺得noop不好,因為系統沒那個條件不做elevator(有人建議,若disk device有很好的request 管理機制,可以用noop);deadline,為因read優先於write ,或許因為這樣測不出好數據(實際上的XGW.STK.K1應該有read也有write );anticipatory或許因為XGW.STK.K1在本例中幾乎都只有write,所以有好的數據;所以筆者覺得cfq配合高的nr_requests較保險,也較能預期對實際狀況(有read也有write)下能有不錯的效能提升
  
   3.system call的分析 
     一般系統調完了,大概也知道程式本身的一些特性,其實程式本身也可以進行分析,提供給開發人員,或者也能夠對系統效率有些改善

nr_requests=4096配合anticipatory
[root@HTS099 queue]# ps -ef | grep XG
root      5909     1 17 21:12 ?        00:00:02 ./XGateway -d
root      5937  4663  0 21:12 pts/2    00:00:00 grep XG
[root@HTS099 queue]# strace -p 5909 -c
Process 5909 attached - interrupt to quit
Process 5909 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 73.46    0.610550           0   5683975           write
 19.98    0.166032           1    112149         1 poll
  1.59    0.013212           0   1858339           gettimeofday
  1.37    0.011416           0    836665           lseek
  1.12    0.009291         107        87           close
  0.71    0.005923           0    154532      5137 read
  0.59    0.004914           0     56046           send
  0.58    0.004855           0    337399           time
  0.49    0.004064           0    112092     56047 recv
  0.06    0.000508           3       185        25 open
  0.02    0.000144           2        90           munmap
  0.02    0.000136           3        53        53 connect
  0.00    0.000036           0       542           brk
  0.00    0.000017           0        53           socket
  0.00    0.000013           0       152           getsockopt
  0.00    0.000000           0        10         8 mkdir
  0.00    0.000000           0         1         1 sigreturn
  0.00    0.000000           0         1           mremap
  0.00    0.000000           0        91           mmap2
  0.00    0.000000           0         1           madvise
  0.00    0.000000           0       110           fcntl64
  0.00    0.000000           0         2           futex
  0.00    0.000000           0         2           tgkill
  0.00    0.000000           0         4         2 accept
  0.00    0.000000           0         4           getsockname
  0.00    0.000000           0         4           getpeername
  0.00    0.000000           0        51           setsockopt
------ ----------- ----------- --------- --------- ----------------
100.00    0.831111               9152640     61274 total

nr_requests=4096配合cfq
[root@HTS099 queue]# strace -p 5552 -c
Process 5552 attached - interrupt to quit
Process 5552 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 87.57    1.342217           0   5684176           write
  8.93    0.136850           1    112181         1 poll
  0.85    0.012984           0   1858717           gettimeofday
  0.71    0.010894           0    836729           lseek
  0.58    0.008904          95        94           close
  0.42    0.006392           0    153624      4182 read
  0.35    0.005406           0     56046           send
  0.33    0.005123           0    337500           time
  0.21    0.003238           0    112092     56047 recv
  0.02    0.000311           2       189        29 open
  0.01    0.000132           2        60        60 connect
  0.01    0.000099          50         2           futex
  0.00    0.000075           1        88           munmap
  0.00    0.000026           0       524           brk
  0.00    0.000020           0        60           socket
  0.00    0.000012           0       167           getsockopt
  0.00    0.000000           0        10         8 mkdir
  0.00    0.000000           0         1         1 sigreturn
  0.00    0.000000           0         1           mremap
  0.00    0.000000           0        91           mmap2
  0.00    0.000000           0       124           fcntl64
  0.00    0.000000           0         2           tgkill
  0.00    0.000000           0         4         2 accept
  0.00    0.000000           0         4           getsockname
  0.00    0.000000           0         4           getpeername
  0.00    0.000000           0        57           setsockopt
------ ----------- ----------- --------- --------- ----------------
100.00    1.532683               9152547     60330 total

ramdisk配合anticipatory
[root@HTS099 queue]# ps -ef | grep XG
root      6260     1  7 21:18 ?        00:00:01 ./XGateway -d
root      6301  4663  0 21:18 pts/2    00:00:00 grep XG
[root@HTS099 queue]# strace -p 6260 -c
Process 6260 attached - interrupt to quit
Process 6260 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 42.12    0.495718           3    154737      5362 read
 38.00    0.447302           0   5683978           write
 15.87    0.186833           2    112153         1 poll
  1.20    0.014139           0   1858164           gettimeofday
  0.88    0.010348           0    836666           lseek
  0.74    0.008721         101        86           close
  0.48    0.005594           0     56046           send
  0.41    0.004834           0    337403           time
  0.28    0.003326           0    112092     56047 recv
  0.01    0.000118           1       186        26 open
  0.00    0.000033           1        53        53 connect
  0.00    0.000022           0       155           getsockopt
  0.00    0.000018           0        53           socket
  0.00    0.000009           0       523           brk
  0.00    0.000008           0        91           mmap2
  0.00    0.000004           0       110           fcntl64
  0.00    0.000000           0        10         8 mkdir
  0.00    0.000000           0        88           munmap
  0.00    0.000000           0         1         1 sigreturn
  0.00    0.000000           0         1           mremap
  0.00    0.000000           0         4           futex
  0.00    0.000000           0         2           tgkill
  0.00    0.000000           0         4         2 accept
  0.00    0.000000           0         4           getsockname
  0.00    0.000000           0         4           getpeername
  0.00    0.000000           0        50           setsockopt
------ ----------- ----------- --------- --------- ----------------
100.00    1.177027               9152664     61500 total


ramdisk配合cfq
[root@HTS099 queue]# ps -ef | grep XG
root      6608     1 13 21:24 ?        00:00:01 ./XGateway -d
root      6638  4663  0 21:24 pts/2    00:00:00 grep XG
[root@HTS099 queue]# strace -p 6608 -c
Process 6608 attached - interrupt to quit
Process 6608 detached
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
 41.36    0.354313           2    153443      4036 read
 32.29    0.276602           0   5683977           write
 20.06    0.171828           2    112155         1 poll
  1.79    0.015369           0    836666           lseek
  1.69    0.014454           0   1858390           gettimeofday
  1.04    0.008922         104        86           close
  0.68    0.005864           0     56046           send
  0.58    0.004947           0    337403           time
  0.38    0.003217           0    112092     56047 recv
  0.09    0.000764           4       186        26 open
  0.01    0.000094           1        88           munmap
  0.01    0.000094           2        53        53 connect
  0.00    0.000035           0       524           brk
  0.00    0.000027           0       155           getsockopt
  0.00    0.000012           0        91           mmap2
  0.00    0.000011           0       110           fcntl64
  0.00    0.000000           0        10         8 mkdir
  0.00    0.000000           0         1         1 sigreturn
  0.00    0.000000           0         1           mremap
  0.00    0.000000           0         1           futex
  0.00    0.000000           0         1           tgkill
  0.00    0.000000           0        53           socket
  0.00    0.000000           0         4         2 accept
  0.00    0.000000           0         4           getsockname
  0.00    0.000000           0         4           getpeername
  0.00    0.000000           0        50           setsockopt
------ ----------- ----------- --------- --------- ----------------
100.00    0.856553               9151594     60174 total

     
補充指令：
     sysctl -w vm.drop_caches=3
     filefrage -v /path to file
     cat /proc/[pid]/status (see "thread"), man 5 proc
     cat /proc/#/status
     cat /proc/interrupt
     cat /proc/partitions (major:minor)
     lspci -v查出硬體與IRQ
     SATA: smartctl -a /dev/sda查詢硬碟資訊
             smartctl -a /dev/sda -d ata"
     dmesg | more 也可以查出一些硬體資訊

     監控目前HDD有在IO作業的PROCESS是哪一支

     while true; do date; ps auxf | awk '{if($8=="D") print $0;}'; sleep 1; done

     sleep 1若改usleep 300000, 則0.3秒監控一次,可以更快反應

     其中,可以取得PID , 進一步透過/proc/PID/io 來找出相對應的read/write統計資料

       rchar => This is the total reads caused by the process in bytes. This includes bytes read from the disk or from console

              wchar => The same as rchar but it accounts for bytes written instead of read

              read_bytes => Actual number of bytes read from the storage

              write_bytes => Actual bytes written to the storage

              Here's the script that shows you the top 5 processes which are reading data from disk:

           grep read_bytes /proc/[0-9]*/io | sort -nrk2 | head -5

             writing data, execute the following:

           grep write_bytes /proc/[0-9]*/io | sort -nrk2 | head -5

             There's also an utility called "iotop" which does the same thing in real time and gives you better top-like output

           yum install iotop

     /proc/PID/status, 則是方面的統計內容

參考資料:
http://yoshinorimatsunobu.blogspot.com/2009/04/linux-io-scheduler-queue-size-and.html

http://www.softpanorama.org/Commercial_linuxes/performance_tuning.shtml

process用top監控的方便

一般用top監控,如果想要留記錄來比較,可能會覺得不太方便,以下是可以留一個時間跟資料內容來當成追蹤記錄
top -S -b -p $(pidof XGateway) grep -e XGateway -e average
[結果]
top - 16:13:11 up 42 days, 21:00, 4 users, load average: 1.11, 1.19, 1.17
31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:12.54 XGateway
top - 16:13:14 up 42 days, 21:00, 4 users, load average: 1.10, 1.18, 1.17
31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:12.54 XGateway
top - 16:13:17 up 42 days, 21:00, 4 users, load average: 1.41, 1.25, 1.19
31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:12.54 XGateway
top - 16:13:20 up 42 days, 21:00, 4 users, load average: 1.41, 1.25, 1.19
31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:12.54 XGateway
top - 16:13:23 up 42 days, 21:00, 4 users, load average: 1.46, 1.26, 1.20

[root@RH20 cts]# cat top.sh#!/bin/bash
pidd=$(pidof XGateway)
while [ 1 ]
do

#Solaris不適用
echo "$(date +%H:%M:%S) $(top -n1 -S -b -p $pidd | grep -e XGateway )"
usleep 500000
done

[結果]17:01:19 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:20 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:21 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:22 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:23 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:24 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:25 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:26 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:27 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:28 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:29 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway
17:01:30 31555 cts 15 0 227m 60m 8416 S 0.0 1.6 0:15.84 XGateway

[root@RH20 cts]# cat vmstat.sh#!/bin/bash
#pidd=$(pidof XGateway)
while [ 1 ]
do
str="$(vmstat 1 2| tail -n 1)"

#在solaris這行要改成 "$(vmstat 1 2 | tail -1l)"

echo "$(date +%H:%M:%S) $str"
done

[結果][root@RH20 cts]# ./vmstat.sh
20:21:02 0 0 896 1011492 325056 2152676 0 0 0 0 1006 4532 2 3 96 0 0
20:21:03 0 0 896 1011616 325056 2152676 0 0 0 260 1059 4558 1 2 97 0 0
20:21:04 0 0 896 1011616 325056 2152684 0 0 0 0 1004 4759 1 2 97 0 0
20:21:05 2 0 896 1011616 325056 2152684 0 0 0 0 1010 2977 1 1 98 0 0


[root@RH20 cts]# cat iostat.sh
#==================================================
# you must install sysstat-* rpm for iostat
#==================================================

#iostat有不同版本,取得字串的行數就會不一樣
iostatver=$(iostat -V &> /dev/stdout | grep sysstat)
if [ "$iostatver" != "" ]; then
ntail=2
nhead=1
else
ntail=1
nhead=1
fi 

if [ "$1" != "" ]; then
device="$1"
else
device="sda"
fi

while [ 1 ]
do
strdate=$(date +%H:%M:%S)
#strstr=$(iostat -x $1 $2 $3 $4 $5 1 2 | tail -n $ntail | head -n $nhead | awk 'NR==1{ printf " %s %s\n","########",$0} NR>=2{printf " %s %s\n","        ",$0}' | sed 's/########/'${strdate}'/')
strstr=$(iostat -x $device 3 2 | tail -n $ntail | head -n $nhead | awk 'NR==1{ printf " %s %s\n","########",$0} NR>=2{printf " %s %s\n","        ",$0}' | sed 's/########/'${strdate}'/')
echo "$strstr"

done

[結果]RH20 /home/cts>./iostat.sh sda
 13:58:58 sda        0     1    0.0    3.0     0.0    16.0    5.3   0.0    0.0   0.0   0
 13:58:59 sda        0     0    0.0    0.0     0.0     0.0    0.0   0.0    0.0   0.0   0
 13:59:00 sda        0    33    0.0   12.0     0.0   192.0   16.0   0.2   18.9   2.0   2

ftp & scp commands via auto-scripts

FTP
(1) Linux type I
[root@HTS099 Downloads]# ftp -vin yyy.yyy.yyy.yyy << EOF
> user [userid] [userpass] [userid]@[ip or hostname]
> binary
> get index.html
> bye
> EOF

Connected to yyy.yyy.yyy.yyy (xxx.xxx.xxx.xxx).
220 Microsoft FTP Service
331 Password required for [userid].
230 User [userid] logged in.
202 ACCT command not implemented.
200 Type set to I.
local: index.html remote: index.html
227 Entering Passive Mode (xxx,xxx,xxx,xxx,23,40).
125 Data connection already open; Transfer starting.
226 Transfer complete.
18893 bytes received in 0.161 secs (117.56 Kbytes/sec)
221

(2) Linux type II
[root@HTS099 Downloads]# { echo "user [userid] [userpass]"
> echo "binary"
> echo "get index.html"
> echo "bye"
> } | ftp -ivn [ip or hostname]

Connected to yyy.yyy.yyy.yyy (xxx.xxx.xxx.xxx).
220 Microsoft FTP Service
331 Password required for [userid].
230 User [userid] logged in.
202 ACCT command not implemented.
200 Type set to I.
local: index.html remote: index.html
227 Entering Passive Mode (xxx,xxx,xxx,xxx,23,40).
125 Data connection already open; Transfer starting.
226 Transfer complete.
18893 bytes received in 0.161 secs (117.56 Kbytes/sec)
221

(3) Linux type III
[root@HTS099 Downloads]# vi ~/.netrc

    machine [IP or hostname] login [userid] password [userpass]
 macdef init
 binary
 get index.html
 bye

#最後一行，記得留空白行，不然ftp時可能會出現錯誤
Connected to cab.org.tw (203.69.42.82).
220 Microsoft FTP Service
Macro definition missing null line terminator.

[root@HTS099 Downloads]# chmod 600 ~/.netrc
＃這裡如果沒改permission，ftp時會出現錯誤
Connected to cab.org.tw (203.69.42.82).
220 Microsoft FTP Service
Error - .netrc file not correct permissions.
Remove password or correct mode (should be 600).

[root@HTS099 Downloads]# ftp cab.org.tw #這裡如果敲錯，就會問你ID跟密碼
Connected to yyy.yyy.yyy.yyy (xxx.xxx.xxx.xxx).
220 Microsoft FTP Service
331 Password required for [userid].
230 User [userid] logged in.
202 ACCT command not implemented.
200 Type set to I.
local: index.html remote: index.html
227 Entering Passive Mode (xxx,xxx,xxx,xxx,23,40).
125 Data connection already open; Transfer starting.
226 Transfer complete.
18893 bytes received in 0.161 secs (117.56 Kbytes/sec)
221

(4) Windows type
C:\Documents and Settings\Administrator>type cab.txt
open yyy.yyy.yyy.yyy
user ＃user在遇到windows的FTP SERVER時，要省略，不然無法成功
[userid]
[userpass]
binary
get index.html
bye
C:\Documents and Settings\Administrator>ftp -i -i -s:"cab.txt"
ftp> open yyy.yyy.yyy.yyy
Connected to yyy.yyy.yyy.yyy.
220 Microsoft FTP Service
User (yyy.yyy.yyy.yyy:(none)):
331 Password required for [userid].

230 User [userid] logged in.
ftp> binary
200 Type set to I.
ftp> get index.html
200 PORT command successful.
150 Opening BINARY mode data connection for index.html(18893 bytes).
226 Transfer complete.
ftp: 18893 bytes received in 0.22Seconds 86.67Kbytes/sec.
ftp> bye
221

＃完整BATCH檔建議如下：get_xxx.bat
ECHO Ftp ready to download data
ftp -i -i -s:"cab.txt" >> mlog.log
move *.log "C:\mlog\"

SCP
shell>cat get_xxx.sh
#!/usr/bin/expect

expect -c "
#spawn ssh -L 1234:172.10.10.1:22 guest@10.1.1.2
spawn scp guest@xxx.xxx.xxx:/index.html
set timeout 1800
expect
{
\"password:\" {send \"[password for guest]\r\";}
}
expect eof;"

build a phplist all-in-one disc iso

[root@HTS099 ~]# mkdir -p /burn/phplist-centos
[root@HTS099 ~]# cd /burn


＃可能會有某些檔案在Linux Copy會有錯誤，建議可以在Windows上COPY下來後再傳過來
[root@HTS099 burn]# cp -R /media/CENTOS55\ i386\ DVD/* /burn/phplist-centos/.
[root@HTS099 phplist-centos]# cd /burn/phplist-centos/
[root@HTS099 phplist-centos]# ls -la /media/CENTOS_5.5\ i386\ DVD/.
./ ../ .discinfo .treeinfo
[root@HTS099 phplist-centos]# cp /media/CENTOS_5.5\ i386\ DVD/.discinfo /burn/phplist-centos/.
[root@HTS099 phplist-centos]# cp /media/CENTOS_5.5\ i386\ DVD/.treeinfo /burn/phplist-centos/.
製作安裝光碟需要將isolinux移出來
[root@HTS099 phplist-centos]# cp -R isolinux /burn/.

＃建立這個目前主要是擺放此次要額外安裝的相關檔案原始壓縮檔
[root@HTS099 phplist-cab]# mkdir extras

＃複製原始phpMyAdmin的壓縮檔到extras
[root@HTS099 html]# ls -l /root/Downloads/phpMyAdmin-2.11.11.3-all-languages.tar.gz
-rw-r--r--. 1 root root 4316983 Mar 17 13:33 /root/Downloads/phpMyAdmin-2.11.11.3-all-languages.tar.gz
[root@HTS099 html]# cp /root/Downloads/phpMyAdmin-2.11.11.3-all-languages.tar.gz /burn/phplist-centos/extras/.

＃複製原始phplist的壓縮檔到extras（備份用）
[root@HTS099 phplist-cab]# ls -l ~/Downloads/phplist*
-rw-r--r--. 1 root root 2317926 Mar 14 11:34 /root/Downloads/phplist-2.10.13.tgz
[root@HTS099 phplist-cab]# cp /root/Downloads/phplist-2.10.13.tgz /burn/phplist-centos/extras/.

#複製phplist自動安裝相關所需的設定檔跟壓縮檔到extras(建置用）
[root@HTS099 ~]#cd /burn/phplist-centos/extras
[root@HTS099 ~]#cp -R /extras /burn/phplist-centos/.
[root@HTS099 ~]#ls -l /burn/phplist-centos/
-rw-r--r--. 1 root root 129 Apr 17 11:24 creation.sql [mysql 權限設定］
-rw-r--r--. 1 root root 103903 Apr 17 11:24 libmcrypt-2.5.7-4.el4.centos.i386.rpm ［phpMyAdmin安裝必要套件］
-rw-r--r--. 1 root root 92788 Apr 17 11:24 libmcrypt-devel-2.5.7-4.el4.centos.i386.rpm ［phpMyAdmin安裝必要套件］
-rw-r--r--. 1 root root 176419 Apr 17 11:24 libmhash-0.9.1-1.2.el5.rf.i386.rpm ［phpMyAdmin安裝必要套件］
-rw-r--r--. 1 root root 88202 Apr 17 11:24 mcrypt-2.6.8-1.el5.i386.rpm ［phpMyAdmin安裝必要套件］
-rwxr-xr-x. 1 root root 40890 Apr 17 11:24 migration.sh [安裝光碟自動安裝後，postscript執行安裝phplist及phpMyAdmin必要scripts檔］
-rw-r--r--. 1 root root 2317926 Apr 17 11:24 phplist-2.10.13.tgz ［phplist安裝包］
-rw-r--r--. 1 root root 5206 Apr 17 11:24 phplist.cnf ［給migration.sh執行時的設定檔，內存phplist必要設定］
-rw-r--r--. 1 root root 993136 Apr 17 11:24 php-mbstring-5.1.6-3.el4s1.10.i386.rpm ［phpMyAdmin安裝必要套件］
-rw-r--r--. 1 root root 15115 Apr 17 11:24 php-mcrypt-5.1.6-1.el4.centos.i386.rpm ［phpMyAdmin安裝必要套件］
-rw-r--r--. 1 root root 4316983 Apr 17 11:24 phpMyAdmin-2.11.11.3-all-languages.tar.gz ［phpMyAdmin安裝包］
-rw-r--r--. 1 root root 3642 Apr 17 11:24 sysproc.php ［phplist新增systool選單功能的網頁］
-rw-r--r--. 1 root root 3278 Apr 17 11:24 systool.php ［phplist新增systool選單功能的網頁］


#製作kickstart file
[root@HTS099 ~]# cp /root/ks.cfg /burn/phplist-centos/ks.cfg
#
# Kickstart configuration file
#

#linux ks=cdrom:/ks.cfg ipaddr=172.16.43.144 gateway=172.16.43.1 netmask=255.255.255.0 ListRoot=[pageroot] StoreAcc=[ftpuser] StoreIP=[ftphost] StorePass=[ftppass] SecureType=gmail
#
#after installation , you could access the page via
# http://[172.16.43.144]/[pageroot]/lists
# the administrator could maintance the server via
# http://[172.16.43.144]/[pageroot]/lists/admin
# user: admin , default password: phplist
#
# This helps the RedHat anaconda installation program to automatically
# configure and install Linux on your machine.
#

# set the language to US english
#lang en_US
key --skip
lang en_US.UTF-8
langsupport en_US
keyboard us

# default is a general 3 button ps/2 mouse
mouse generic3ps/2

# set the timezone to EDT
#timezone America/New_York
timezone Asia/Taipei --utc

# set a root password -- you can also encrypt this
#rootpw _some_password_
rootpw redhat
#rootpw --iscrypted $1$lvU6kOK5$wzGy5BlTgkfyXjFO8C0Qo0
selinux --disabled
firewall --disabled

# uncomment to have the system reboot after install has completed
reboot

# install in text mode
text

# use the Lilo boot loader, turn on the linear flag, and write it into
# the master boot record
#bootloader --useLilo --linear --location=mbr
bootloader --location=mbr --driveorder=sda --append="rhgb quiet"


# uncomment to have the user confirms each dialog
#interactive

# this is like interactive but the user confirms each page not each dialog
#autostep

# install Linux, don't upgrade
install
# install from the cdrom
cdrom

# clear all of the existing partitions from the disk and make new ones
clearpart --all --initlabel
part /boot --fstype ext3 --size 52
part swap --size 1024
part / --fstype ext3 --size 10240 --grow

# You can configure network settings here. I am working toward using
# dhcp an pump to get the configs for a certain host and then fix them
# statically at the end of the installation process.
#network --bootproto static --ip ... --netmask ... --gateway ... --nameserver ...
#network --bootproto static --ip 172.16.43.144 --netmask 255.255.255.0 --gateway 172.16.43.1 --nameserver 168.95.1.1

# configure how passwords are stored
auth --useshadow --enablemd5

# setup a medium strength firewall and allow ssh incoming connections
#firewall --medium --ssh

# skip the x windows configuration
skipx

#
# The following lists the RPMs that will be installed by the system.
# The @ lines are sort-of macros which are defined in the
# RedHat/base/comps file on the cdrom. I've added nc and zsh at the
# end because I wanted them and they weren't in any of the @ lines.
#
# An alternative to this list would be to list the specific RPMs to
# install although that might get big. You will need to check this
# list carefully since we are going to pair down the RPMs so they fill
# on one CD.
#
%packages
#@X Window System
#@Network Support
#@Spelling
#@NFS File Server
#@DNS Name Server
#@Network Managed Workstation
#@Emacs
#@Utilities
#@Software Development
#nc
#zsh
#@office
#@development-libs
@editors
#@gnome-software-development
@text-internet
#@x-software-development
#@gnome-desktop
#@dialup
@core
@base
#@games
@java
@base-x
#@graphics
@web-server
#@printing
#@sound-and-video
#@admin-tools
#@development-tools
#@graphical-internet
@mysql
#mesa-libGLU-devel
#kexec-tools
#device-mapper-multipath
#xorg-x11-utils
#xorg-x11-server-Xnest
#xorg-x11-server-Xvfb
#libsane-hpaio
#imake
php
php-mysql
#for phpMyAdmin
php-common
php-mbstring
php-imap
-sysreport
#
# The following commands are executed at the end of the installation
# process by /bin/sh while you are chrooted into the new system. This
# allows you to add user accounts, turn on or off autobooted software,
# etc..
#
# Note that at the end of the script I have an example of how to get
# input from the user. I'd like to hear if anyone has better ways of
# doing this. My way basically uses the 3rd virtual terminal to both
# echo to and read from. It works.
#

%post --nochroot
# cdrom is exist at /tmp/cdrom when "--nochroot"
mkdir -p /mnt/sysimage/mnt/source
mount -t iso9660 /tmp/cdrom /mnt/sysimage/mnt/source
mkdir -p /mnt/sysimage/extras
cp -R /mnt/sysimage/mnt/source/extras/* /mnt/sysimage/extras/.

%post
# Replace with ftaylor's (working) code.
#eval $(cat /proc/cmdline | awk '{print $NF}')
eval $(sed 's/ /\n/g' /proc/cmdline | grep '^ipaddr=')
eval $(sed 's/ /\n/g' /proc/cmdline | grep '^gateway=')
eval $(sed 's/ /\n/g' /proc/cmdline | grep '^netmask=')
eval $(sed 's/ /\n/g' /proc/cmdline | grep '^dns=')
#$Recovery=mysql backup main file name in ftp server that you would like D/L
#eval $(sed 's/ /\n/g' /proc/cmdline | grep '^Recovery=')
#$LISTROOT=phplist web page root folder
eval $(sed 's/ /\n/g' /proc/cmdline | grep '^ListRoot=')
#ftp account infomation
eval $(sed 's/ /\n/g' /proc/cmdline | grep '^StoreIP=')
eval $(sed 's/ /\n/g' /proc/cmdline | grep '^StoreAcc=')
eval $(sed 's/ /\n/g' /proc/cmdline | grep '^StorePass=')
eval $(sed 's/ /\n/g' /proc/cmdline | grep '^SecureType=')

if [ "$SecureType" == "" ]; then
SecureType="simple"
fi

if [ "$ListRoot" == "" ]; then
ListRoot="phplist"
fi


mkdir -p /extras/recovery
mkdir -p /extras/backup

chmod 775 /extras
chmod 775 /extras/recovery
chmod 775 /extras/backup
chown root.apache /extras
chown root.apache /extras/recovery
chown root.apache /extras/backup


# turn off packages that we don't want running
#echo Turning off some packages
#/sbin/chkconfig --level 0123456 apmd off
#/sbin/chkconfig --level 0123456 sendmail off
#/sbin/chkconfig --level 0123456 lpd off
#/sbin/chkconfig --level 0123456 xfs off
#/sbin/chkconfig --level 0123456 gpm off
#/sbin/chkconfig --level 0123456 wine off

# turning on packages that we need
#echo Turning on some packages
#/sbin/chkconfig --level 345 ntpd on

# Maybe add a ntp server to the ntp.conf
#echo Adding entries to ntp.conf
#cat >> /etc/ntp.conf << EOF
#server my.good.ntp.home.com
#EOF
# Maybe add some entries to the sudoers file
#echo Adding some entries to the sudoers file
#cat >> /etc/sudoers << EOF
#me ALL = (ALL) ALL
#EOF

# example of user i/o if necessary
#
# # redirect IO to the 3rd terminal -- thanks Charles.
# exec < /dev/tty3 > /dev/tty3
#
# echo -n "Enter something: "
# read yn
# echo "We got $yn"

###############################################################################
# network setting #
###############################################################################
echo "DEVICE=eth0" > /etc/sysconfig/network-scripts/ifcfg-eth0
echo "ONBOOT=yes" >> /etc/sysconfig/network-scripts/ifcfg-eth0
echo "IPADDR=$ipaddr" >> /etc/sysconfig/network-scripts/ifcfg-eth0
echo "NETMASK=$netmask" >> /etc/sysconfig/network-scripts/ifcfg-eth0
echo "GATEWAY=$gateway" >> /etc/sysconfig/network-scripts/ifcfg-eth0
#chattr +i /etc/sysconfig/network-scripts/ifcfg-eth0

echo "NETWORKING=yes" > /etc/sysconfig/network
echo "NETWORKING_IPV6=no" >> /etc/sysconfig/network
echo "HOSTNAME=PHPLIST" >> /etc/sysconfig/network
#chattr +i /etc/sysconfig/network

echo "search example.com" > /etc/resolv.conf
if ["$dns" = ""]; then
echo "nameserver 168.95.1.1" >> /etc/resolv.conf
else
echo "nameserver $dns" >> /etc/resolv.conf
fi
#chattr +i /etc/resolv.conf


###############################################################################
# copy phplist and build it #
###############################################################################


#setup phplist
chkconfig mysqld on
chkconfig httpd on
service mysqld start


tar -zxvf /extras/phplist-2.10.13.tgz -C /
#cp -f /extras/php.ini /etc/php.ini
ln -s /phplist-2.10.13/public_html /var/www/html/$ListRoot


#/usr/bin/mysqladmin -u root password 'rootcab'
#/usr/bin/mysql -u root --password='rootcab' < /extras/creation.sql

service httpd start

#sudoers
cp /etc/sudoers /etc/sudoers.org
cat /etc/sudoers | sed '/Defaults\x20\x20\x20\x20requiretty/ s/^/#/' > /etc/sudoers

cat >> /etc/sudoers << EOF
Cmnd_Alias BOOTING = /sbin/init
apache ALL=BOOTING,NOPASSWD:ALL
EOF

#setup phpMyAdmin
/bin/tar -zxvf /extras/phpMyAdmin-2.11.11.3-all-languages.tar.gz -C /
ln -s /phpMyAdmin-2.11.11.3-all-languages /var/www/html/phpMyAdmin
rpm -ivh /extras/libmcrypt-2.5.7-4.el4.centos.i386.rpm
rpm -ivh /extras/libmcrypt-devel-2.5.7-4.el4.centos.i386.rpm
rpm -ivh /extras/php-mcrypt-5.1.6-1.el4.centos.i386.rpm
rpm -ivh /extras/libmhash-0.9.1-1.2.el5.rf.i386.rpm
rpm -ivh /extras/mcrypt-2.6.8-1.el5.i386.rpm
#cp /extras/config.inc.php /phpMyAdmin-2.11.11.3-all-languages/.


#restore mysql
#cp -f /extras/storeFROMdb.sh /root/storeFROMdb.sh
#cp -f /extras/recoveryTOcab.sh /root/recoveryTOcab.sh
#if [ "$Recovery" != "" ]; then
# mkdir -p /root/recovery
# openvt -sw -- wget http://www.cab.org.tw/$Recovery.tar.gz -O /root/recovery/$Recovery.tar.gz
# /bin/tar -zxvf /root/recovery/$Recovery.tar.gz -C /root/recovery/
# /usr/bin/mysql -u root --password='rootcab' cab < /root/recovery/$Recovery.sql
## /root/recoveryTOcab.sh $Recovery
#fi

if [ "$StoreIP" != "" ] && [ "$StoreAcc" != "" ] && [ "$StorePass" != "" ]; then
/extras/migration.sh -c=/extras/phplist.cnf -t=$SecureType --listdir=\\\/$ListRoot --ftprtdir=\\\/extras -A=\\\/$ListRoot\\\/lists\\\/admin -H=\\\/$ListRoot\\\/lists --ftpuser=$StoreAcc --ftphost=$StoreIP --ftppass=$StorePass --ftpmode=binary > /root/migration.log
###############################################################################
# cron job #
###############################################################################
cat << EOF >> /etc/crontab
0 * * * * root /usr/sbin/ntpdate -b time.stdtime.gov.tw;/sbin/hwclock --systohc --utc
0 0 * * * root /extras/storeFROMdb.sh
EOF
else
/extras/migration.sh -c=/extras/phplist.cnf -t=$SecureType --listdir=\\\/$ListRoot --ftprtdir=\\\/extras -A=\\\/$ListRoot\\\/lists\\\/admin -H=\\\/$ListRoot\\\/lists >/root/migration.log

###############################################################################
# cron job #
###############################################################################
cat << EOF >> /etc/crontab
0 * * * * root /usr/sbin/ntpdate -b time.stdtime.gov.tw;/sbin/hwclock --systohc --utc
EOF
fi

###############################################################################
# rc.local #
###############################################################################

echo "/usr/sbin/ntpdate -b time.stdtime.gov.tw;/sbin/hwclock --systohc --utc" >> /etc/rc.local
# echo "0 18 * * * /root/storeFROMcab.sh root rootcab $StoreIP $StoreAcc $StorePass" >> /etc/rc.local

###############################################################################
# shutdown daemons #
###############################################################################
chkconfig apmd off
chkconfig bluetooth off
chkconfig cups off
chkconfig nfslock off
chkconfig pcscd off
chkconfig xfs off
chkconfig sendmail off


#copy a file for new menuitem in phplist
#cp -f /extras/systool.php /var/www/html/$ListRoot/lists/admin
#cp -f /extras/sysproc.php /var/www/html/$ListRoot/lists/admin


#準備燒錄
[root@HTS099 phplist-cab]# cp -rf /burn/phplist-centos/isolinux /burn/.

[root@HTS099 burn]# cd /burn
[root@HTS099 burn]# mkisofs -A "CENTOS_2-10-13_i386_DVD" -V "CENTOS_2-10-13_i386_DVD" -J -R -v -T -o phplist-centos.iso -b isolinux/isolinux.bin -c isolinux/boot.cat -no-emul-boot -boot-load-size 4 -boot-info-table phplist-centos

：
：
Writing: Directory tree cleanup Start Block 534
Done with: Directory tree cleanup Block(s) 0
Writing: Extension record Start Block 534
Done with: Extension record Block(s) 1
Writing: The File(s) Start Block 535
0.30% done, estimate finish Thu Mar 17 13:47:11 2011
0.60% done, estimate finish Thu Mar 17 13:49:58 2011
0.89% done, estimate finish Thu Mar 17 13:49:02 2011
1.19% done, estimate finish Thu Mar 17 13:48:34 2011
1.49% done, estimate finish Thu Mar 17 13:48:18 2011
：
：
Total translation table size: 884504
Total rockridge attributes bytes: 389387
Total directory bytes: 602112
Path table size(bytes): 268
Done with: The File(s) Block(s) 1679275
Writing: Ending Padblock Start Block 1679810
Done with: Ending Padblock Block(s) 150
Max brk space used 3a5000
1679960 extents written (3281 MB)
[root@HTS099 burn]#
[root@HTS099 burn]# ll
total 3359932
drwxr-xr-x. 2 root root 4096 Mar 17 13:44 isolinux
drwxr-xr-x. 12 root root 4096 Mar 17 13:40 phplist-centos
-rw-r--r--. 1 root root 3440558080 Mar 17 13:52 phplist-centos.iso

＃安裝使用
把ISO燒成光碟後，開機，可以發現熟悉的RHEL5開機
輸入：linux ks＝cdrom:/ks.cfg ipaddr=...... netmask=.... gateway=.... ....
就會依剛才設定所分配的內容來安裝


＃參考資料
http://ms.ntcb.edu.tw/~steven/article/install_dvd.htm
http://www.centos.org/docs/5/html/5.2/Installation_Guide/s1-kickstart2-startinginstall.html