Sean的吃喝玩樂+工作: Fuse

這次修改原本的安裝，多數以rpm packages安裝為主
(1)系統環境準備
    此次使用的環境，使用為virtual machine來實現
       host : CentOS-6.0
       kernel: 2.6.32-71.el6.x86_64 #1 SMP
       MemTotal:        7891424 kB(/proc/meminfo)
       CPU(Quad core)      : Intel(R) Core(TM) i3-2130 CPU @ 3.40GHz
   HDD： 1TB＊2 （LVM ＝＝＞ 2T）


       guest: CentOS-6.0
       kernel: 2.6.32-71.el6.x86_64 #1 SMP
       MemTotal:        1G
       CPU *1
   HDD： 100GB

       KFS的環境使用virtual machine的NAT環境，也把dhcp打開
       guest總共佈建4台，1台metaserver, 3台chunkserver
       這4台/etc/hosts設定如下
   192.168.65.1 mserver    #yum/ftp server
   192.168.65.150 t01nn    #metaserver
   192.168.65.151 t01d1    #chunkserver1
   192.168.65.152 t01d2    #chunkserver2
   192.168.65.153 t01d3    #chunkserver3
(2)系統安裝
使用kickstart檔來進行初期的部署(4台guest都照此安裝)，所有rpm在這個階段都安裝完成，簡單描述幾個重點
：
    selinux --disabled
        firewall --disabled
        ：

        bootloader --location=mbr --driveorder=sda --append="rhgb quiet"
        ：
#在host,我們佈建了vsftpd以及httpd，
        #預設在/var/ftp/pub，/var/www/html(soft link to /var/ftp/pub)
        install
        url --url=http://192.168.65.1/pub
        :
        clearpart --drives=sda --all --initlabel
        part / --asprimary --fstype="ext4" --ondisk=sda --grow --size=1024
        part /boot --asprimary --fstype="ext4" --ondisk=sda --size=200
        part swap --fstype="swap" --ondisk=sda --recommended
        :

        #這裡將要安裝的套件一併在下面一一指定安裝
        %packages
        @core
        @server-policy
        ftp
        telnet
        ntpdate
        gcc
        gcc-c++
        make
        cmake
        log4cpp
        boost
        boost-devel
        expat
        openssl-devel
        java-1.6.0-openjdk
        java-1.6.0-openjdk-devel
        uuid
        uuid-devel
        libuuid
        libuuid-devel
        ant
        python
        python-devel

        fuse-devel
        fuse-libs
        %end

        %post
        #接下來的設定，可依個人喜好，把系統預設安裝的設定項，在這裡一併完成
        #筆者在這裡接連把下列的設定一併完成
        # 1. /etc/sysconfig/network-scripts/ifcfg-eth0 設定網卡
        # 2. /etc/sysconfig/network   設定hostname
#    /etc/hosts
#    /etc/resolv.conf
# 3. 安裝xfsprogs相關套件，由於預設CentOS 6.0的安裝光碟沒有，於是下載後
        #      置入yum/ftp server上，在%post時補充安裝
        #      xfsprogs-3.1.1-7.el6.x86_64.rpm
        #      xfsprogs-devel-3.1.1-7.el6.x86_64.rpm

        # 4. kfs-0.5.tar.gz的安裝 --> /usr/local/kfs-0.5
        #      詳細安裝可見［KFS］(KosmosFS) 安裝方法一
        # 5. /usr/local/kfs-0.5/scripts/machines.cfg
        # 6. /usr/local/kfs-0.5/conf/machines.txt
        # 7. /usr/local/kfs-0.5/conf/ChunkServer.prp
        # 8. Fuse掛載KFS
        #      詳細安裝可見KFS mount by Fuse
# 9. 設定方便的scripts, 可將kfs setup, start, stop及mount等指令設置成scripts
        # 10. 更新套件，如果在這個步驟啟動KFS，fuse mount KFS時，
        #       可能會發生coredump。有可能的原因Centos 6.0是預設rpm套件太舊。
        #       筆者經過以下套件更新後，KFS mount即恢復正常(沒有一一試過是哪個太舊）
        #       如果不想輸入這麼長的一串更新套件，可以直接連上Centos 預設的yum server
        #       進行yum update即可
        #        rpm -Uvh ~/pkgs/boost-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-date-time-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-devel-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-filesystem-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-graph-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-iostreams-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-program-options-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-python-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-regex-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-serialization-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-signals-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-system-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-test-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-thread-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/boost-wave-1.41.0-11.el6_1.2.x86_64.rpm \
        #                       ~/pkgs/cpp-4.4.6-4.el6.x86_64.rpm \
        #                       ~/pkgs/expat-2.0.1-11.el6_2.x86_64.rpm \
        #                       ~/pkgs/fuse-devel-2.8.3-4.el6.x86_64.rpm \
        #                       ~/pkgs/fuse-libs-2.8.3-4.el6.x86_64.rpm \
        #                       ~/pkgs/gcc-4.4.6-4.el6.x86_64.rpm \
        #                       ~/pkgs/gcc-c++-4.4.6-4.el6.x86_64.rpm \
        #                       ~/pkgs/libblkid-2.17.2-12.7.el6.x86_64.rpm \
        #                       ~/pkgs/libgcc-4.4.6-4.el6.x86_64.rpm \
        #                       ~/pkgs/libgomp-4.4.6-4.el6.x86_64.rpm \
        #                       ~/pkgs/libstdc++-4.4.6-4.el6.x86_64.rpm \
        #                       ~/pkgs/libstdc++-devel-4.4.6-4.el6.x86_64.rpm \
        #                       ~/pkgs/libuuid-2.17.2-12.7.el6.x86_64.rpm \
        #                       ~/pkgs/libuuid-devel-2.17.2-12.7.el6.x86_64.rpm \
        #                       ~/pkgs/make-3.81-20.el6.x86_64.rpm \
        #                       ~/pkgs/ntpdate-4.2.4p8-2.el6.centos.x86_64.rpm \
        #                       ~/pkgs/openssl-1.0.0-20.el6_2.5.x86_64.rpm \
        #                       ~/pkgs/openssl-devel-1.0.0-20.el6_2.5.x86_64.rpm \
        #                       ~/pkgs/python-2.6.6-29.el6_2.2.x86_64.rpm \
        #                       ~/pkgs/python-devel-2.6.6-29.el6_2.2.x86_64.rpm \
        #                       ~/pkgs/python-libs-2.6.6-29.el6_2.2.x86_64.rpm \
        #                       ~/pkgs/telnet-0.17-47.el6.x86_64.rpm \
        #                       ~/pkgs/util-linux-ng-2.17.2-12.7.el6.x86_64.rpm \
        #                       ~/pkgs/java-1.6.0-openjdk-1.6.0.0-1.48.1.11.3.el6_2.x86_64.rpm \
        #                       ~/pkgs/java-1.6.0-openjdk-devel-1.6.0.0-1.48.1.11.3.el6_2.x86_64.rpm

(3)系統啟動
   系統啟動需要再執行以下作業
   1）ssh password-less
利用這篇ssh password-less session config的做法，將t01nn, t01d1, t01d2, t01d3這幾台都設定為彼此不需要密碼登入
     2）在t01d1, t01d2, t01d3都執行以下指令
           rsync -av /usr/local/kfs-0.5/ t01d1:/usr/local/kfs-0.5/
           rsync -av /usr/local/kfs-0.5/ t01d2:/usr/local/kfs-0.5/
           rsync -av /usr/local/kfs-0.5/ t01d3:/usr/local/kfs-0.5/
    3）setup your KFS
         cd /usr/local/kfs-0.5/scripts
python kfssetup.py -f machines.cfg -m ../conf/machines.txt -b ../build -w ../webui
    4) start up your KFS
          python /usr/local/kfs-0.5/scripts/kfslaunch.py -f /usr/local/kfs-0.5/scripts/machines.cfg -m /usr/local/kfs-0.5/conf/machines.txt -s

    5) mount your KFS
         /usr/local/kfs-0.5/build/bin/kfs_fuse /mnt/kfs -f -o allow_other

後記：

目前的安裝方法，己經縮短安裝時間，約20分鐘即可在Virtual Machine將這4台KFS架設完成，但為求日後便利：

1）可以製作便利的ssh password-less scripts, 日後在主機異動時，可以快速設定

2) 可以製作便利的server-cnf-update scripts, 便利日後主機異動時，更新設定

machines.txt machines.cfg ChunkServer.prp

rsync...

3）自製kfs rpm，讓安裝更快速便利（目前kfs安裝約要5分鐘左右）且更適合大量部署

assumed you already installed KFS, and KFS could be started as well. Then you could keep follow few steps below for mount it. KFS setup please refer to［KFS］(KosmosFS) 安裝方法一
(1)stop metaservers and chunkservers
(2)yum install fuse-devel.x86_64 fuse-libs.x86_64 -y
(3)make sure CMakeLists.txt setting
    rpm -ql fuse-devel
   /lib64/libfuse.so
   /lib64/libulockmgr.so
   /usr/include/fuse
   /usr/include/fuse.h
   /usr/include/fuse/cuse_lowlevel.h
   /usr/include/fuse/fuse.h
   :
   :

    vi /usr/local/kfs-0.5/CMakeLists.txt
    # Fill in the path to fuse library
    SET(Fuse_LIBRARY_DIR "/lib64")          #go to line 136,137, XX_DIR should be the same with what rpm(fuse...rpm) you just installed
    SET(Fuse_INCLUDE_DIR "/usr/include")
(4)re-build KFS
    cd /usr/local/kfs-0.5/build
    cmake -D CMAKE_BUILD_TYPE=Release /usr/local/kfs-0.5/
or
    cmake -D CMAKE_BUILD_TYPE=RelWithDebInfo /usr/local/kfs-0.5/
    gmake
    gmake install

#kfs_fuse will be built in /usr/local/kfs-0.5/build/bin

    cd /usr/local/kfs-0.5/scripts
    python kfssetup.py -f machines.cfg -m ../conf/machines.txt -b ../build -w ../webui

#start you KFS
    python kfslaunch.py -f machines.cfg -m ../conf/machines.txt -s
(5)build a new prp file for fuse mounting

  look at the code src/cc/libkfsClient/KfsClient.cc:
  KfsClientPtr
  KfsClientFactory::GetClient(const char *propFile)
  {
      bool verbose = false;
  #ifdef DEBUG
      verbose = true;
  #endif
      if (theProps().loadProperties(propFile, '=', verbose) != 0) {
          KfsClientPtr clnt;
     return clnt;
      }

      return GetClient(theProps().getValue("metaServer.name", ""),
                       theProps().getValue("metaServer.port", -1));

  }

cd /usr/local/kfs-0.5/build/bin
vi kfs.prp #it should be named kfs.prp compulsorily
metaServer.name = metaserver    #not metaServer.host
    metaServer.port = 20000
(6)mount KFS
cd /usr/local/kfs-0.5/build/bin
    mkdir -p /mnt/kfs
    ./kfs_fuse /mnt/kfs -f -o allow_other
    # -f : force the system to run the process in foreground
    # -o allow_other : allow other user to access the mount point

mount
:
    /dev/mapper/vg_01-lv_home on /home type ext4 (rw)
    none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
    kfs_fuse on /mnt/kfs type fuse.kfs_fuse (rw,nosuid,nodev,allow_other)

    ls -l /mnt/kfs
    total 0
    d--------- 0 root root    0 2012-07-25 11:00 dumpster
    d--------- 0 root root 9987 2012-07-26 15:12 hypertable

(7) FAQ
     1) Q: kfs_fuse: /usr/include/boost/smart_ptr/shared_ptr.hpp:418: T* boost::shared_ptr< >::operator->() const [with T = KFS::KfsClient]: Assertion `px != 0' failed.
        A: look at your kfs.prp(exist ? typo ?), make sure metaServer.name (not metaServer.host)

2) Q: fuse: bad mount point `/mnt/kfs': Transport endpoint is not connected

A: if you had fixed all config files , but error again. Try to restart kfs service

umount /mnt/kfs

          cd /usr/local/kfs-0.5/scripts
            python kfslaunch.py -f machines.cfg -m ../conf/machines.txt -S
            python kfslaunch.py -f machines.cfg -m ../conf/machines.txt -s
          cd /usr/local/kfs-0.5/build/bin
            ./kfs_fuse /mnt/kfs -f -o allow_other

reference :   http://yaronspace.cn/blog/archives/799

Sean的吃喝玩樂+工作

2012年8月10日星期五

［KFS］(KosmosFS) 安裝方法二

2012年7月26日星期四

KFS mount by Fuse

文章分類

網誌存檔

關於我自己