upgrading hypertable under cdh4

(0) environmental
     Hostname  processes          ip
     ========  =================  ================
     master01  Hypertable.Master  192.168.122.11
               Hyperspace
               DfsBroker
               RangeServer
               Namenode(CDH4)      
     smaster   SecondaryNamenode  192.168.122.10
               JobTracker
     knode020  DfsBroker          192.168.122.20
               RangeServer
               Datanode
               TaskTracker

     knode021  DfsBroker          192.168.122.21
               RangeServer
               Datanode
               TaskTracker

     old Hypertable Version: 

              hypertable-0.9.6.5.59309c4-linux-x86_64.tar.gz

     new Hypertable Version: 

              hypertable-0.9.7.7-linux-x86_64.rpm 

(1) download 0.9.7.7 rpm file
   ssh to master01 as root user:
   [root@master01~] cd /mnt/nfssvr/extras/hypertable/rpm
   [root@master01 rpm]# wget http://cdn.hypertable.com/packages/0.9.7.7/hypertable-0.9.7.7-linux-x86_64.rpm

(2) upgrading
     (a) stop your hypertable cluster(my Capfile is in /root)
     [root@master01 rpm] cd ~
     [root@master01 rpm] cap stop
     [root@master01 rpm] ps -ef | grep hyper
   # if there was any process exist, stop it

     [root@master01 rpm] /opt/hypertable/current/bin/stop-servers.sh

     (b) modify your Capfile
     [root@master01 rpm] cd ~
     [root@master01 ~] vi Capfile
         :

         #set :hypertable_version, "0.9.6.5.59309c4"
         set :hypertable_version, "0.9.7.7"
         #set :default_pkg, "/mnt/nfssvr/extras/hypertable/rpm/hypertable-0.9.6.5.59309c4-linux-x86_64.tar.gz"
         set :default_pkg, "/mnt/nfssvr/extras/hypertable/rpm/hypertable-0.9.7.7-linux-x86_64.rpm"

         :

         :

     (c) install it & upgrade it

     [root@master01 ~] cd ~

     [root@master01 ~] cap install_package

     [root@master01 ~] cap dist

     [root@master01 ~] ls /opt/hypertable

drwxr-xr-x 7 root root      4096 Jun  7 16:55 0.9.6.5.59309c4
drwxr-xr-x 8 root root      4096 Jun 10 14:05 0.9.7.7
lrwxrwxrwx 1 root root        15 Jun  7 16:55 current -> 0.9.6.5.59309c4
-rwxr-xr-x 1 root root 103614419 Jun  7 16:55 hypertable-0.9.6.5.59309c4-linux-x86_64.tar.gz

   #so far , /opt/hypertabl/0.9.7.7 was installed 

   #,but soft link "current" was still pointed to 0.9.6.5.59309c4

   #cap dist to update new version

     [root@master01 ~] cap upgrade

   #"cap upgrade" whill :

   # 1.verifying the format is Compatible (between old hypertable and new one). Also you could verify compatible by yourself "/opt/hypertable/current/upgrade-ok.sh"

   # 2.Copy folders(run/ conf/ run/...etc)from old(0.9.6.5.59309c4) version to new one

   # 3.redirect the soft link to "0.9.7.7"

     [root@master01 ~] ls /opt/hypertable

drwxr-xr-x 7 root root      4096 Jun  7 16:55 0.9.6.5.59309c4

drwxr-xr-x 7 root root      4096 Jun 10 14:06 0.9.7.7

lrwxrwxrwx 1 root root         7 Jun 10 14:06 current -> 0.9.7.7

-rwxr-xr-x 1 root root 103614419 Jun  7 16:55 hypertable-0.9.6.5.59309c4-linux-x86_64.tar.gz

      (d)soft link

      #disable  softlink liblog4cpp

     [root@master01 ~] vi /mnt/nfssvr/extras/lib_alias.sh

      :

      :

  #if [ ! -e /opt/hypertable/current/lib/liblog4cpp.so ]; then
  #    log4cpp_fname="$(ls /opt/hypertable/current/lib/liblog4cpp* | grep liblog4cpp.so.)"
  #    ln -s $log4cpp_fname       /opt/hypertable/current/lib/liblog4cpp.so

  #fi
  exit 0

    

     #relink lib alias

     [root@master01 ~] cap lib_alias

     [root@master01 ~]# ls -l /opt/hypertable/current/lib/*.so

           :

     lrwxrwxrwx 1 root root       45 2013-06-13 13:48 /opt/hypertable/current/lib/libevent.so -> /opt/hypertable/current/lib/libevent-1.4.so.2

           :

           :

           :

     lrwxrwxrwx 1 root root       46 2013-06-13 13:38 /opt/hypertable/current/lib/libthrift.so -> /opt/hypertable/current/lib/libthrift-0.8.0.so

     (e) start hypertable

     #Now you could start up hypertable normally. Under CDH4 , you need to do more before you startup (otherwise you would get a exception)

     

   [root@master01 ~]# cap shell
   cap> /opt/hypertable/current/bin/set-hadoop-distro.sh cdh4
    [establishing connection(s) to master01, knode020, knode021, smaster]
     ** [out :: knode020] Hypertable successfully configured for Hadoop cdh4
     ** [out :: knode021] Hypertable successfully configured for Hadoop cdh4
     ** [out :: master01] Hypertable successfully configured for Hadoop cdh4
     ** [out :: smaster] Hypertable successfully configured for Hadoop cdh4
    cap> quit
    exiting

   [root@master01 ~]# cap start

     (f) for makefile

   CC=g++
   CFLAGS=-c -Wall -D_REENTRANT -I/opt/hypertable/current/include \
          -I/opt/hypertable/current/include/thrift
   LDFLAGS=-rdynamic -L/opt/hypertable/current/lib -lHyperThrift \
           -lHyperCommon -lHypertable -lthrift -levent
   #        -lHyperCommon -lHypertable -lthrift -levent -llog4cpp
   BIN=../bin

   all: client_test candle candle2

   client_test: client_test.o
           $(CC) client_test.o $(LDFLAGS) -o client_test
           cp client_test $(BIN)

           :

           :

   client_test.o: client_test.cc
           $(CC) $(CFLAGS) client_test.cc

           :

           :

   clean:
           rm -rf *o client_test

           :

           :

see more:

     http://hypertable.com/documentation/administrator_guide/upgrading

    https://groups.google.com/forum/?fromgroups#!topic/hypertable-user/iuduinCcIjU

  

Implement REST API Querying by iphone from HDFS and Hypertable

Service Preparation

 172.16.43.X         172.16.43.145                           192.168.122.X
                        +-----+                     +--------+    |
   O              50070 |     | <-----------> 50070 |master01| ---+
  -+-   --------> 50075 | pen |               9000  +--------+    |
   |              10080 |     | <------+             @VM/Guest    |
  / \                   +-----+        |                          |
 iphone-3GS              @HOST         |            +--------+    |
                                       +----> 50075 |knode???| ---+
                                                 80 +--------+    | hypertable's REST 80 port 
                                                     @VM/Guest
REST API test by curl command line from PC

[List files]
curl -i "http://172.16.43.145:50070/webhdfs/v1/tmp/?op=LISTSTATUS"
    172.16.43.X   ==> OK
    172.16.43.145 ==> OK

[Read a file]
curl -i -L "http://172.16.43.145:50070/webhdfs/v1/tmp/Capfile3?op=OPEN"
    172.16.43.X   ==> X
    172.16.43.145 ==> OK

curl -i -L "http://172.16.43.145:50075/webhdfs/v1/tmp/Capfile3?op=OPEN&namenoderpcaddress=master01:9000"
    172.16.43.X   ==> OK
    172.16.43.145 ==> OK

[Create and Write a file]
curl X PUT -L "http://172.16.43.145:50070/webhdfs/v1/tmp/Capfile5?op=CREATE" -T /root/anaconda-ks.cfg
    172.16.43.X   ==> X
    172.16.43.145 ==> OK

curl -i -X PUT -L "http://172.16.43.145:50075/webhdfs/v1/tmp/Capfile4?op=CREATE&namenoderpcaddress=192.168.122.11:9000" -T ./index.html
    172.16.43.X   ==> OK
    172.16.43.145 ==> OK

[Overwrite a file]
curl -i -X PUT -L "http://172.16.43.145:50070/webhdfs/v1/tmp/Capfile3?op=CREATE&overwrite=true" -T /root/anaconda-ks.cfg
    172.16.43.X   ==> X
    172.16.43.145 ==> OK

curl -i -X PUT -L "http://172.16.43.145:50075/webhdfs/v1/tmp/Capfile3?op=CREATE&namenoderpcaddress=master01:9000&overwrite=true" -T ./index.html
    172.16.43.X   ==> OK
    172.16.43.145 ==> OK

[Query hypertable "RS_METRICS" within "sys" database]
curl -i "http://172.16.43.145:10080/rest/api-ht.php?rquest=scanner"
    172.16.43.X   ==> OK
    172.16.43.145 ==> OK


[Append a line to a file]
## !!! NOT implmentment here
##reference : http://blog.cloudera.com/blog/2009/07/file-appends-in-hdfs/
##                http://docs.fluentd.org/articles/out_webhdfs#parameters
##hdfs-site.xml setting
  <property>
    <name>dfs.webhdfs.enabled</name>
    <value>true</value>
  </property>

  <property>
    <name>dfs.support.append</name>
    <value>true</value>
  </property>

  <property>
    <name>dfs.support.broken.append</name>
    <value>true</value>
  </property>

Client Testing Tool

Xcode : 4.6
iphone: 3GS (iOS6.1) or simulator
libcurl: 7.30.0

[import libcurl into testRest project]
download and uncompress curl-7.30.0( from http://home.comcast.net/~seiryu/software/ioscurl-7.30.0.tar.gz)
[user@testRest_witCurl]mkdir libcurl
[user@testRest_witCurl]cp ~/Downloads/ioscurl-7.30.0.tar
[user@testRest_witCurl]cd libcurl
[user@libcurl] tar -xvf ioscurl-7.30.0.tar

Open testRest_withCurl\testRest.xcodeproj by Xcode4.6
Project -> TARGETS -> testRest -> Build Phases -> Link Binary With Libraries -> Add Item(+)
  (1) Add Other...
      libcurl/libcurl-device.a
      libcurl/libcurl-sumulator.a
  (2) Add
      CFNetwork.framework
      Security.framework
      libz.dylib

   
[testRest_withCurl/AppDelegate.h]
//
//  AppDelegate.h
//  testRest
//
//  Created by kway on 13/4/22.
//  Copyright (c) 2013年 kway. All rights reserved.
//

#import <UIKit/UIKit.h>

typedef enum {
    actGET = 1,
    actPOST,
    actPUT,
    actDELETE
} Actions;


@interface AppDelegate : UIResponder <UIApplicationDelegate>
{
    BOOL isLocation; // -L
    BOOL isUpload;   // -T

    Actions action;  // -X GET
}

-(size_t) showRest:(char *)data onlysize:(size_t)size;

@property (strong, nonatomic) IBOutlet UIWindow *window;
@property (strong, nonatomic) IBOutlet UITextField *textIP;
@property (strong, nonatomic) IBOutlet UITextField *textPort;
@property (strong, nonatomic) IBOutlet UITextView *tvURI;
@property (strong, nonatomic) IBOutlet UITextView *tvRest;
@property (strong, nonatomic) IBOutlet UISegmentedControl *scType;

-(IBAction)scTypeChangeHandler:(id)sender;
-(IBAction)btnSend;



@end
//global function for callback when curl
size_t handle(char * data, size_t size, size_t count, void * ptr2AppDelegate);


[testRest_withCurl/AppDelegate.m]
//
//  AppDelegate.m
//  testRest
//
//  Created by kway on 13/4/22.
//  Copyright (c) 2013年 kway. All rights reserved.
//

#import "AppDelegate.h"
#import "../libcurl/iOScURL/curl/curl.h"



@implementation AppDelegate



- (void)dealloc
{
    [_window release];
    [_tvRest release];
    [_tvURI release];
    [_textIP release];
    [_textPort release];
    [_scType release];
    [super dealloc];
}
-(void) initApp{
    action=actGET;
    isLocation=false;
    isUpload=false;
    _tvRest.text = @"";
    _textIP.text=@"172.16.43.145";
    _textPort.text=@"50070";
    _tvURI.text=[NSString stringWithFormat:@"http://%@:%@/webhdfs/v1/tmp/?op=LISTSTATUS",_textIP.text,_textPort.text];
}
-(IBAction)scTypeChangeHandler:(id)sender{
//    int selectedIndex=[self.scType selectedSegmentIndex];
    int selectedIndex=[sender selectedSegmentIndex];
    switch(selectedIndex)
    {
        case 0: // DIR
            action=actGET;
            isLocation=false;
            isUpload=false;
            _textPort.text=@"50070";
            _tvURI.text=[NSString stringWithFormat:@"http://%@:%@/webhdfs/v1/tmp/?op=LISTSTATUS",_textIP.text,_textPort.text];
            break;
        case 1: // CAT
            action=actGET;
            isLocation=true;
            isUpload=false;
            _textPort.text=@"50075";
            _tvURI.text=[NSString stringWithFormat:@"http://%@:%@/webhdfs/v1/tmp/Capfile?op=OPEN&namenoderpcaddress=master01:9000",_textIP.text,_textPort.text];
            break;
        case 2: //SCANNER
            action=actGET;
            isLocation=false;
            isUpload=false;
            _textPort.text=@"10080";
            _tvURI.text=[NSString stringWithFormat:@"http://%@:%@/rest/api-ht.php?rquest=scanner",_textIP.text,_textPort.text];
            break;
    }
}

-(size_t) showRest:(char *)data onlysize:(size_t)size
{
    if(size >0)
    {
        _tvRest.text=[NSString stringWithFormat:@"%s",data];
    }
    else
    {
        _tvRest.text=@"";
    }
    return size;
}
-(IBAction)btnSend
{
    curl_global_init(CURL_GLOBAL_ALL);
    CURL *curl;
    CURLcode res;
    curl = curl_easy_init();
    if(curl!=NULL)
    {
        curl_easy_setopt(curl,CURLOPT_WRITEFUNCTION,handle);
        curl_easy_setopt(curl,CURLOPT_WRITEDATA,self);
        curl_easy_setopt(curl, CURLOPT_URL,[_tvURI.text UTF8String]);
        if(isLocation) // for -L option
        {
            curl_easy_setopt(curl,CURLOPT_FOLLOWLOCATION,1);
        }
        res = curl_easy_perform(curl);
        curl_easy_cleanup(curl);
    }
}
- (BOOL)application:(UIApplication *)application didFinishLaunchingWithOptions:(NSDictionary *)launchOptions
{
//    self.window = [[[UIWindow alloc] initWithFrame:[[UIScreen mainScreen] bounds]] autorelease];
    // Override point for customization after application launch.
//    self.window.backgroundColor = [UIColor whiteColor];

    [self.window makeKeyAndVisible];
    [self initApp];
    return YES;
}

- (void)applicationWillResignActive:(UIApplication *)application
{
    // Sent when the application is about to move from active to inactive state. This can occur for certain types of temporary interruptions (such as an incoming phone call or SMS message) or when the user quits the application and it begins the transition to the background state.
    // Use this method to pause ongoing tasks, disable timers, and throttle down OpenGL ES frame rates. Games should use this method to pause the game.
}

- (void)applicationDidEnterBackground:(UIApplication *)application
{
    // Use this method to release shared resources, save user data, invalidate timers, and store enough application state information to restore your application to its current state in case it is terminated later.
    // If your application supports background execution, this method is called instead of applicationWillTerminate: when the user quits.
}

- (void)applicationWillEnterForeground:(UIApplication *)application
{
    // Called as part of the transition from the background to the inactive state; here you can undo many of the changes made on entering the background.
}

- (void)applicationDidBecomeActive:(UIApplication *)application
{
    // Restart any tasks that were paused (or not yet started) while the application was inactive. If the application was previously in the background, optionally refresh the user interface.
}

- (void)applicationWillTerminate:(UIApplication *)application
{
    // Called when the application is about to terminate. Save data if appropriate. See also applicationDidEnterBackground:.
}

@end

//global function for callback when curl
size_t handle(char * data, size_t size, size_t count, void * ptr2AppDelegate)
{
    if(ptr2AppDelegate != nil)
        [ (AppDelegate *)ptr2AppDelegate showRest:data onlysize:size*count];
    return size*count;
}

Result

Reference:
http://home.comcast.net/~seiryu/libcurl-ios.html
http://curl.haxx.se/libcurl/c/allfuncs.html

Implementation of REST API for Hypertable

reference: http://www.9lessons.info/2012/05/create-restful-services-api-in-php.html



192.168.122.11     192.168.122.10
+------------+     +-------------+
|master01    |     |smaster      |
|            |     |             |
|            |     |             |-+
|            |     |             | |
+------+-----+     +-------+-----+ |          +----NAT(@KVM)
       |                   |      NFS:        |
-------+-------------------+------------------+
       |                   |          
+------+-----+     +-------+-----+   
|knode020    |     |knode021     |   
|php & httpd |     |php & httpd  |   
|            |     |             |   
|            |     |             |   
+------------+     +-------------+   
192.168.122.20     192.168.122.21    

1. CDH4 installation 

   skipped

2. Hypertable 0.9.6.4 installlation

   skipped
   the binary package for cdh4 : https://groups.google.com/forum/?fromgroups=#!msg/hypertable-user/dmQhx_i7HfQ/YE32E9Omv3EJ

3. php and httpd preparation

      [root@knode020 rest]# yum install php mysql mysql-server php-mysql httpd
      ** mysql* is used for trace and debug, php and httpd is just work for REST API for HT

      [root@knode020 rest]#  chkconfig httpd on
      [root@knode020 rest]#  service httpd start

4. REST API php code for Hyptertable


    the most important source in api-ht.php as follow:

 class API extends REST {

  public $data = "";
  
  const DB_SERVER = "localhost";
  const DB = "sys";
  
  private $db = NULL;
  private $ns = NULL;

  public function __construct(){
   parent::__construct();
   $this->dbConnect(); 
  }
  public function __destruct(){
   if($this->ns)
    $this->db->namespace_close($this->ns);
  }
  
  private function dbConnect(){
   $this->db = new Hypertable_ThriftClient(self::DB_SERVER, 38080);
   if($this->db)
    $this->ns = $this->db->namespace_open(self::DB);

  }
  
  public function processApi(){
   $func = strtolower(trim(str_replace("/","",$_REQUEST['rquest'])));
   if((int)method_exists($this,$func) > 0)
    $this->$func();
   else
    $this->response('',404);    
  }
  
  
  
  private function scanner(){ 
   if($this->get_request_method() != "GET"){
    $this->response('',406);
   }
   $scanner = $this->db->scanner_open($this->ns, "RS_METRICS", new Hypertable_ThriftGen_ScanSpec(array('rot_limit' => 1000)));

   $cells = $this->db->scanner_get_cells_as_arrays($scanner);
   if(empty($cells)){
    $this->response('', 204); // If no records "No Content" status
   }

   $reault = array();
   foreach($cells as $cell)
   {
    print_r(json_encode($cell));
    $result[] = $cell;
   }
   $this->response($this->json($result), 200);
  }

  private function hql(){ 
   if($this->get_request_method() != "GET"){
    $this->response('',406);
   }
   $limit = $this->_request['limit'];  
   if($limit > 0)
    $cells = $this->db->hql_query_as_arrays($this->ns, "select * from RS_METRICS limit $limit");
   else
    $cells = $this->db->hql_query_as_arrays($this->ns, "select * from RS_METRICS ");
   if(empty($cells->cells)){
    $this->response('', 204); // If no records "No Content" status
   }

   $reault = array();
   foreach($cells->cells as $cell)
   {
    $result[] = $cell;
   }
   $this->response($this->json($result), 200);
  }

  private function json($data){   if(is_array($data)){
    return json_encode($data);
   }
  }
 }

5. test by curl

      [root@knode020 rest]#  curl -i "http://localhost/rest/api-ht.php?rquest=hql&limit=2"
      [root@knode020 rest]#  curl -i http://localhost/rest/api-ht.php?rquest=hql
      [root@knode020 rest]#  curl -i http://localhost/rest/api-ht.php?rquest=scanner

Appendix A: mysql REST API

Query User
[root@knode020 rest]# curl -i "http://localhost/rest/api.php?rquest=users"
HTTP/1.1 200 OK
Date: Wed, 23 Jan 2013 01:44:31 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.3.3
Content-Length: 166
Connection: close
Content-Type: application/json

[{"user_id":"1","user_fullname":"Sean","user_email":"seanlo0805@gmail.com"},{"user_id":"2","user_fullname":"Jewel","user_email":"testjewel@yahoo.com.tw"}]

Login User
[root@knode020 rest]# curl -i -X POST -d "email=seanlo0805@gmail.com" -d "pwd=sean" http://localhost/rest/api.php?rquest=login
HTTP/1.1 200 OK
Date: Wed, 23 Jan 2013 03:37:05 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.3.3
Content-Length: 83
Connection: close
Content-Type: application/json

{"user_id":"1","user_fullname":"Sean","user_email":"seanlo0805@gmail.com"}

Delete User
root@knode020 rest]# curl -i -X DELETE "http://localhost/rest/api.php?rquest=deleteUser&id=1"
HTTP/1.1 200 OK
Date: Wed, 23 Jan 2013 05:40:34 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.3.3
Content-Length: 61
Connection: close
Content-Type: application/json

{"status":"Success","msg":"Successfully one record deleted."}

[root@knode020 rest]# curl -i "http://localhost/rest/api.php?rquest=users"
HTTP/1.1 200 OK
Date: Wed, 23 Jan 2013 05:41:14 GMT
Server: Apache/2.2.15 (CentOS)
X-Powered-By: PHP/5.3.3
Content-Length: 82
Connection: close
Content-Type: application/json

[{"user_id":"2","user_fullname":"Jewel","user_email":"testjewel@yahoo.com.tw"}]

Appendix B: trace and debug REST API for hadoop

[QUERY]
curl -i "http://master01:50070/webhdfs/v1/hypertable?op=LISTSTATUS"


curl -i "http://master01:50070/webhdfs/v1/hypertable?op=GETFILESTATUS"

curl -i "http://master01:50070/webhdfs/v1/hypertable?op=GETHOMEDIRECTORY"

[CREATE FILE]
curl -i -X PUT "http://master01:50070/webhdfs/v1/hypertable/abc.txt?op=CREATE&user.name=root"
curl -i -X PUT "http://knode020:50075/webhdfs/v1/hypertable/abc.txt?op=CREATE&namenoderpcaddress=master01:9000&user.name=root"

[OPEN FILE]
curl -i -L "http://knode020:50075/webhdfs/v1/hypertable/abc.txt?op=OPEN&user.name=root&namenoderpcaddress=master01:9000"
curl -i -L "http://master01:50070/webhdfs/v1/hypertable/abc.txt?op=OPEN"

Appendix C: json 2 xml

<?php
function array_to_xml(array $arr, SimpleXMLElement $xml)
{
    foreach ($arr as $k => $v) {
        is_array($v)
            ? array_to_xml($v, $xml->addChild($k))
            : $xml->addChild($k, $v);
    }
    return $xml;
}

if (!isset($GLOBALS['THRIFT_ROOT']))
        $GLOBALS['THRIFT_ROOT'] = "/opt/hypertable/current/lib/php";


require_once $GLOBALS['THRIFT_ROOT'].'/ThriftClient.php';

$client = new Hypertable_ThriftClient("localhost", 38080);
$namespace = $client->namespace_open("sys");

//array('revs'=> 1);

$scanner = $client->scanner_open($namespace, "RS_METRICS", new
Hypertable_ThriftGen_ScanSpec(array('rot_limit' => 1000)));

$cells = $client->scanner_get_cells_as_arrays($scanner);

@header("Content-Type: text/xml");
$result=array();
foreach($cells as $cell)
{

        $result[] = $cell;
}
print array_to_xml($result, new SimpleXMLElement('<root/>'))->asXML();

$client->namespace_close($namespace);
?>

基於mpich2進行雲端運算實例

近來為了一個實驗性的架構，以支應明年進行服務擴充而準備。選定了以Hadoop+Hypertable為基底的資料服務；並選用了MPICH2+Torque來進行分散式運算的工作。在這裡先不論其好壞(日後再論述)，先進行實作之探討：

主機架構

       選用的Hypertable+Hadoop在先前的文章裡已有描述其安裝與建置步驟；而MPICH2+Torque的建置過程也在可以這裡找到。建置過程，礙於成本上考量，每一台主機皆是以虛擬機來達成多台的部署。在MPICH2+Torque的架構上，選用2台電腦做為分散式計算的compute nodes，其中一台為主要分散式運算服務的發送指令主機(Head node)
        在這個案例裡，利用Client Agent(cagent)在compute node上來當作服務客戶要求的程序(process)，採multiprocess方式運作，每當一個用戶端連線時，隨即在compute node上fork出一個程式來服務它。後續再依用戶端的要求，對2台compute node進行相對應服務的取得(分散運算或雲端運算)

        上圖，即是針對上述的概念所進行一項實作內容的雛形架構。服務用戶端的程式為cagent(Client Agent)，而用戶端連線後，想要取得計算移動平均線的服務mv_avg ( moving average)。 從流程的概念上分述如下：
(1)用戶端以TCP/IP對Client Agent取得連線的同意後，Client Agent於是fork另一程式出來進行單獨服務
(2)Client Agent依用戶請求，應取得移動平均之計算結果(mv_avg)。Client Agent依此，向內部的運算節點取得運要之資源(此例，透過mpiexec將mv_avg程序分成多節點計算)
(3)mv_avg以MPICH2為基，可以分別提供3種計算方式，(A)單節點運算方式，(B)雙節點運算方式(即將工作分派于2個節點平均計算)，(C)多節點計算方式(即以第0節點來發號施令，其餘節點依其派于工作進行計算)
(4)各節點交付計算任務後(計算5、10、15、20、30、40、60、80、120、180、240)等移動平均計算，各計算節點所需之資料來源，乃是透Hypertable Thrift API向Hypertable取得，各計算計後將結果交付第0節點
(5)第0節點依計算結果透過Client Agent，一一回覆給用戶端

以下為Mpi工作分配的主要源碼：


/*
mode-1

  [single node] --+
             ^    | task id == 0 , do it all
             |    |
             +----+

mode-2

  do #0,2,4...tasks
  [taskid == 0] ---+   +---[taskid == 1]
           ^  ^    |   |  do #1,3,5.... tasks
           |  |    |   |
           |  +----+   |
           +-----------+
             
mode-3

  [taskid == 0]          +--->[taskid == 1]
         ^ ^ ^           |  
         | | +-----------+ +->[taskid == 2]
 dispatch| +---------------+
  task # |    
         +------------------->[taskid == n]
                               finish task
                                and
                               reply data [seq][sendbuff=data1,data2,data3]...
         
*/


   if(ntasks == 1) /*mode-1, single node run all tasks*/
   {
       int i;
       for (i=taskid; i >sizeofar(mv_avg_para);i+=ntasks ) { 
 
      if(argc == 2)
          GetCandleDataDefault(sendbuff,baserec,basebuff, &realsize ,baserec,mv_avg_para[i] ,symb); /*from Hypertable*/
      else
          GetCandleDataPeriod(sendbuff,baserec,basebuff, &realsize ,baserec,mv_avg_para[i] ,symb, start_date,end_date); /*from Hypertable*/
      send_channel( channel, sendbuff , realsize,i);
               
       } 
   }
   else if(ntasks == 2) /*mode-2 , 2 nodes share all tasks*/
   {
       int wait=1;
       if( taskid == 0)
       {
           int i,j,count, source;
           for(i=0 ; i <= sizeofar(mv_avg_para) ;i++){
          task2do[i]= TASK_INPIT;
           }
           i=taskid;
           while(wait)
           {
               int flag;
               MPI_Iprobe(MPI_ANY_SOURCE, REPLY, MPI_COMM_WORLD,&flag, &status);
               if(flag) /*data received*/
               {
                   source = status.MPI_SOURCE;
                   MPI_Get_count(&status, MPI_LONG, &count);
                   count = count > baserec ? baserec : count;
                   MPI_Recv(inptbuff, count, MPI_LONG, source, REPLY, MPI_COMM_WORLD, &status);
                   task2do[inptbuff[0] ]=TASK_DONE;
              send_channel( channel, &inptbuff[1] , count-1,inptbuff[0]);

               }
      if ( i < sizeofar(mv_avg_para) ) { 
        
    if(task2do[i ]==TASK_INPIT)
    {
                  task2do[i ]=TASK_RUN;
                  if(argc == 2)
                      GetCandleDataDefault(sendbuff,baserec,basebuff, &realsize ,baserec,mv_avg_para[i] ,symb); /*from Hypertable*/
                  else
                      GetCandleDataPeriod(sendbuff,baserec,basebuff, &realsize ,baserec,mv_avg_para[i] ,symb, start_date,end_date); /*from Hypertable*/
                      
                  send_channel( channel, sendbuff , realsize,i);
                  
                  task2do[i ]=TASK_DONE;
                  i+=ntasks;

              }
               
      } 
      wait=0;
      for (j=0; j < sizeofar(mv_avg_para);j++ ) { 
      wait+=task2do[j];
      } 
           }

       }
       else /*taskid == 1*/
       {
           int i;
  for (i=taskid; i < sizeofar(mv_avg_para);i+=ntasks ) { 
 
          if(argc == 2)
              GetCandleDataDefault(sendbuff,baserec,basebuff, &realsize ,baserec,mv_avg_para[i] ,symb); /*from Hypertable*/
          else
              GetCandleDataPeriod(sendbuff,baserec,basebuff, &realsize ,baserec,mv_avg_para[i] ,symb, start_date,end_date); /*from Hypertable*/
               outpbuff[0] = i;
          if(realsize > 0)
          {
          MPI_Send( outpbuff, realsize+1, MPI_LONG, 0, REPLY, MPI_COMM_WORLD ); 
      }
  } 
       }
   }
   else /*mode-3, multiple nodes tasks ,ntasks > 2*/
   {
    int i,ierr,j,count;
    int wait=1;
    if(taskid==0)  
    {
       for(i=0 ; i <= sizeofar(mv_avg_para) ;i++){
           task2do[i]= TASK_INPIT;
       }
       /*first dispatch task to nodes*/
       for(i=1,j=0 ; i < ntasks && j < sizeofar(mv_avg_para) ;i++,j++){
                    ierr=MPI_Send(&j,1,MPI_INT,
                      i,REQUEST,MPI_COMM_WORLD); 
           task2do[j]= TASK_RUN; 
       }
       
       /*receiving result , and assign next task*/
   while (wait) { 
       int pos;
       MPI_Recv( inptbuff, baserec+1, MPI_LONG, MPI_ANY_SOURCE, REPLY, MPI_COMM_WORLD, &status ); 
                MPI_Get_count(&status, MPI_LONG, &count);
                pos = inptbuff[0];
                
           send_channel( channel, &inptbuff[1] , count-1,inptbuff[0]);
           
       task2do[ pos  ]=TASK_DONE; 
       wait=0;
       for (i=0; i < sizeofar(mv_avg_para);i++ ) { 
           if(task2do[i] == TASK_INPIT)
           {
            /*dispatch next task*/
                        ierr=MPI_Send(&i,1,MPI_INT,
                      status.MPI_SOURCE ,REQUEST,MPI_COMM_WORLD);
               task2do[i]= TASK_RUN; 
           }
       wait+=task2do[i];
       } 
   } 
   
   /*all task was done , send 'finish(==0)' to all nodes*/
       for(i=1,j=0 ; i < ntasks && j < sizeofar(mv_avg_para) ;i++,j++){
           int noCalc=-1;
                    ierr=MPI_Send(&noCalc,1,MPI_INT,
                      i,REQUEST,MPI_COMM_WORLD);  /*all works done!!*/ 
  
       }
    }
    else
    {
       int mv_avg_pos=0;
   MPI_Recv( &mv_avg_pos, 1, MPI_INT, 0, REQUEST, 
    MPI_COMM_WORLD, &status ); 
       while(mv_avg_pos >= 0)
       {
           if(argc == 2)
               GetCandleDataDefault(sendbuff,baserec,basebuff, &realsize ,baserec,mv_avg_para[mv_avg_pos] ,symb); /*from Hypertable*/
           else
               GetCandleDataPeriod(sendbuff,baserec,basebuff, &realsize ,baserec,mv_avg_para[mv_avg_pos] ,symb, start_date,end_date); /*from Hypertable*/
                outpbuff[0] = mv_avg_pos;
           if(realsize > 0)
           {
           MPI_Send( outpbuff, realsize+1, MPI_LONG, 0, REPLY, MPI_COMM_WORLD ); 
       }
       MPI_Recv( &mv_avg_pos, 1, MPI_INT, 0, REQUEST, 
        MPI_COMM_WORLD, &status ); 
   }
        }
   }
   
   MPI_Barrier(MPI_COMM_WORLD);

依上述的計算結果
(1)Hypertable 僅有300筆的情況下的簡單計算
   mode-1：1.397秒 (1 process 1 nodes)

   mode-2：1.296秒 (2 processes 2 nodes)

   mode-3：1.298秒 (3 processes under 2 nodes)

   mode-3：1.216秒 (11 processes under 2 nodes)

(2)模擬各節點需，每個round(每計算一次移動平均)需要耗時1秒以上複雜計算時的結果
   mode-1：11.015秒 (1 process 1 nodes)

   mode-2：8.015秒(2 processes 2 nodes)

   mode-3：10.018秒(3 processes under 2 nodes)

   mode-3：2.031秒(11 processes under 2 nodes)

PS. 模擬耗工作，是以計算完移動平均後，再Sleep 1秒鐘為之。因為是簡單表示工作分派，不作其他資源耗用之假想

以上結果，對於表示的平行運算，只能粗淺意會，大略呈現分散運算的概念，實際需要考量現實世界的因素還有很多，需要加輔經驗的驗證方能有效適用各項計算工作。

實際上，若1 個compute node配置一個CPU或Core，上述的模擬應可大略表達分散運算的目的。在進行分散運算時，仍要注意工作內容是否採分散運算較佳或者集中1個process較佳

(1)compute nodes的 CPU & RAM資源的分佈狀況(是否足以進行分散運算)

(2)service process本身的工作是否能夠進行拆解

(3)service process的工作拆解後，分散於各節點，會不會反而因為網路傳送耗時而失去意義

(4)分散運算有其資源使用與最快速計算時間中，最適當的滿足點。應視運算成本與可容許計算時間之間，取得一個平衡

平台可再擴充之部分
(1)配合PBS進行批次作業之工作分派，以及工作之監控
(2)配合PBS對於常用之Service Process採常駐形式進行服務
(3)用戶與Client Agent間可以用多種介面支援XML, Json並輔以資料加密與壓縮