站長資訊網FastDFS是一個開源的輕量級分布式文件系統，由跟蹤服務器（tracker server）、存儲服務器（storage server）和客戶端（client）三個部分組成，主要解決了海量數據存儲問題，特別適合以中小文件（建議范圍：4KB < file_size <500MB）為載體的在線服務。
（1）每次上傳文件后都會返回一個地址，用戶需要自己保存此地址。
（2）為了支持大容量，存儲節點（服務器）采用了分卷（或分組）的組織方式。存儲系統由一個或多個卷組成，卷與卷之間的文件是相互獨立的，所有卷的文件容量累加就是整個存儲系統中的文件容量。一個卷可以由一臺或多臺存儲服務器組成，一個卷下的存儲服務器中的文件都是相同的，卷中的多臺存儲服務器起到了冗余備份和負載均衡的作用。
配置環境：
CentOS 7.5 x 64
tracker server: 192.168.5.71（tracker暫時一臺，后續再添加一臺做nginx+keepalived高可用）
storage server: 192.168.5.72 192.168.5.73（group1）
storage server: 192.168.5.74 192.168.5.75（group2）
#hosts文件配置
cat >> /etc/hosts << EOF 192.168.5.71 storage71.blufly.com storage71 192.168.5.72 storage72.blufly.com storage72 192.168.5.73 storage73.blufly.com storage73 192.168.5.74 storage74.blufly.com storage74 192.168.5.75 storage75.blufly.com storage75 EOF###———————- 安裝配置tracker server —————————###
yum -y install gcc-c++ libevent#先安裝libfastcommon
cd /opt git clone https://github.com/happyfish100/libfastcommon.git cd libfastcommon/ ./make.sh ./make.sh install
#安裝fastdfs:
cd /opt git clone https://github.com/happyfish100/fastdfs.git cd fastdfs ./make.sh ./make.sh install#配置文件解釋：
tracker.conf //負責均衡調度服務器配置文件 client.conf //客戶端上傳配置文件 http.conf //http服務器配置文件 storage.conf//文件存儲服務器配置文件 mime.types //文件類型配置文件#修改tracker server的配置文件
cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf vi /etc/fdfs/tracker.conf bind_addr= 改為 bind_addr=192.168.25.204 base_path=/home/yuqing/fastdfs 改為 base_path=/data/fastdfs http.server_port=8080 改為 http.server_port=80#啟動tracker server
mkdir -p /data/fastdfs /etc/init.d/fdfs_trackerd start [root@storage71 fastdfs]# netstat -tnlp|grep fdfs tcp 0 0 192.168.5.71:22122 0.0.0.0:* LISTEN 11631/fdfs_trackerd#開機啟動
/sbin/chkconfig --add fdfs_trackerd /sbin/chkconfig --level 2345 fdfs_trackerd on###————————— 安裝配置storage server ———————-###
yum -y install gcc-c++ libevent#先安裝libfastcommon
cd /opt git clone https://github.com/happyfish100/libfastcommon.git cd libfastcommon/ ./make.sh ./make.sh install
#安裝fastdfs:
cd /opt git clone https://github.com/happyfish100/fastdfs.git cd fastdfs ./make.sh ./make.sh install#配置storage server
cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf #修改storage server的配置文件： vm /etc/fdfs/storage.conf bind_addr= 改為 bind_addr=192.168.5.72 #base_path需要和tracker部分的base_path保持一致，如果有修改過tracker的 base_path=/home/yuqing/fastdfs 改為 base_path=/data/fastdfs #修改storage的資源存放路徑 store_path0=/home/yuqing/fastdfs 改為 store_path0=/data/fastdfs #如果有多個掛載磁盤則定義多個store_path，如下 #store_path1=...... #store_path2=...... 修改storage的對應的tracker_server的ip地址和端口 tracker_server=192.168.209.121:22122 改為 tracker_server=192.168.5.71:22122 #如果有多個則配置多個tracker_server #tracker_server=...... #tracker_server=...... http.server_port=8888 改為 http.server_port=80#創建數據目錄，并啟動storage server
mkdir -p /data/fastdfs /etc/init.d/fdfs_storaged start [root@storage72 fastdfs]# netstat -tnlp|grep fdfs tcp 0 0 192.168.5.72:23000 0.0.0.0:* LISTEN 11624/fdfs_storaged#開機啟動
/sbin/chkconfig --add fdfs_storaged /sbin/chkconfig --level 2345 fdfs_storaged on#查看tracker和storage的鏈接情況
[root@storage71 ~]# netstat -tnlpa|grep 22122 tcp 0 0 192.168.5.71:22122 0.0.0.0:* LISTEN 11631/fdfs_trackerd tcp 0 0 192.168.5.71:22122 192.168.5.72:63755 ESTABLISHED 11631/fdfs_trackerd tcp 0 0 192.168.5.71:22122 192.168.5.73:47697 ESTABLISHED 11631/fdfs_trackerd#在tracker server上配置客戶端上傳文件
[root@storage71 fastdfs]# cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf vim /etc/fdfs/client.conf #base_path需要和tracker部分的base_path保持一致 base_path=/home/yuqing/fastdfs 改為 base_path=/data/fastdfs tracker_server=192.168.0.197:22122 改為 tracker_server=192.168.5.71:22122#上傳個文件，如果上傳成功，會返回一個url，我們記住這個url，后面安裝了nginx就可以通URL來訪問上傳的這個文件
[root@storage71 fastdfs]# /usr/bin/fdfs_test /etc/fdfs/client.conf upload /root/test.jpg group_name=group1, ip_addr=192.168.5.72, port=23000 storage_upload_by_filename group_name=group1, remote_filename=M00/00/00/wKgFSFvAem6AfpJXAAA2FXv_nRs556.jpg source ip address: 192.168.5.72 file timestamp=2018-10-12 18:41:50 file size=13845 file crc32=2080349467 example file url: http://192.168.5.72/group1/M00/00/00/wKgFSFvAem6AfpJXAAA2FXv_nRs556.jpg storage_upload_slave_by_filename group_name=group1, remote_filename=M00/00/00/wKgFSFvAem6AfpJXAAA2FXv_nRs556_big.jpg source ip address: 192.168.5.72 file timestamp=2018-10-12 18:41:51 file size=13845 file crc32=2080349467 example file url: http://192.168.5.72/group1/M00/00/00/wKgFSFvAem6AfpJXAAA2FXv_nRs556_big.jpg
通過客戶端連接并上傳文件：
#上傳"/etc/passwd"文件到fastdfs服務器: # fdfs_upload_file /etc/fdfs/client.conf /etc/passwd 下載剛才上傳的文件： # fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/wKgFSFvBpdSAB33ZAAI8IU3BQ48068.jpg 查看文件屬性： # fdfs_file_info /etc/fdfs/client.conf group1/M00/00/00/wKgFSFvBpdSAB33ZAAI8IU3BQ48068.jpg 刪除上傳的文件： # fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKgFSFvBpdSAB33ZAAI8IU3BQ48068.jpg # fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKgFSFvBtTuAGz2VAAAJHTkxfCU312.txt 監控fastdfs的狀態: # fdfs_monitor /etc/fdfs/client.conf
###——————- 在storage server上安裝nginx ————————-###
#安裝nginx關于fastdfs集合的擴展模塊，注意這個模塊nginx默認的編譯模塊中沒有，需手動下載安裝
cd /opt wget http://nginx.org/download/nginx-1.15.5.tar.gz #jemalloc 優化nginx，內存管理 wget https://github.com/jemalloc/jemalloc/releases/download/5.1.0/jemalloc-5.1.0.tar.bz2 wget https://ftp.pcre.org/pub/pcre/pcre-8.42.tar.gz #nginx緩存清理模塊 wget http://labs.frickle.com/files/ngx_cache_purge-2.3.tar.gz git clone https://github.com/happyfish100/fastdfs-nginx-module.gityum -y install openssl openssl-devel zlib zlib-devel glibc glibc-devel glib2 glib2-devel bzip2 bzip2-devel ncurses ncurses-devel curl curl-develtar -jxvf jemalloc-5.1.0.tar.bz2 cd jemalloc-5.1.0 ./configure --prefix=/usr/local/jemalloc --libdir=/usr/local/lib make;make install echo "/usr/local/lib" > /etc/ld.so.conf.d/local.conf ldconfig cd ../ /usr/sbin/groupadd www /usr/sbin/useradd -g www www -s /sbin/nologin tar -zxvf pcre-8.42.tar.gz tar -zxvf ngx_cache_purge-2.3.tar.gz tar -zxvf nginx-1.15.5.tar.gz cd nginx-1.15.5 ./configure --prefix=/usr/local/nginx --with-pcre=/opt/pcre-8.42 --user=www --group=www --with-http_stub_status_module --with-http_ssl_module --with-http_flv_module --with-http_gzip_static_module --with-ld-opt="-ljemalloc" --with-http_realip_module --add-module=/opt/fastdfs-nginx-module/src --add-module=/opt/ngx_cache_purge-2.3 make;make install#修改mod_fastdfs配置文件
cp /opt/fastdfs-nginx-module/src/mod_fastdfs.conf /etc/fdfs/ cd /opt/fastdfs/conf/ #進入fastdfs安裝包文件夾 cp anti-steal.jpg http.conf mime.types /etc/fdfs/ vi /etc/fdfs/mod_fastdfs.conf base_path=/tmp 改為 base_path=/data/fastdfs tracker_server=tracker:22122 改為 tracker_server=192.168.5.71:22122 store_path0=/home/yuqing/fastdfs 改為 store_path0=/data/fastdfs url_have_group_name = false 改為 true#配置nginx
server { listen 80; server_name 192.168.5.72; location /group1/M00 { ngx_fastdfs_module; } }#添加nginx啟動腳本
cat >> /etc/init.d/nginx <<EOF #! /bin/sh ulimit -n 65535 # Description: Startup script for nginx # chkconfig: 2345 55 25 PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin DESC="nginx daemon" NAME=nginx DAEMON=/usr/local/nginx/sbin/$NAME CONFIGFILE=/usr/local/nginx/conf/$NAME.conf PIDFILE=/usr/local/nginx/logs/$NAME.pid SCRIPTNAME=/etc/init.d/$NAME set -e [ -x "$DAEMON" ] || exit 0 do_start() { $DAEMON -c $CONFIGFILE || echo -n "nginx already running" } do_stop() { kill -INT `cat $PIDFILE` || echo -n "nginx not running" } do_reload() { kill -HUP `cat $PIDFILE` || echo -n "nginx can't reload" } case "$1" in start) echo -n "Starting $DESC: $NAME" do_start echo "." ;; stop) echo -n "Stopping $DESC: $NAME" do_stop echo "." ;; reload|graceful) echo -n "Reloading $DESC configuration..." do_reload echo "." ;; restart) echo -n "Restarting $DESC: $NAME" do_stop do_start echo "." ;; *) echo "Usage: $SCRIPTNAME {start|stop|reload|restart}" >&2 exit 3 ;; esac exit 0 EOF#將nginx添加到啟動服務中
chmod 700 /etc/init.d/nginx /sbin/chkconfig --add nginx /sbin/chkconfig --level 2345 nginx on#啟動nginx
[root@storage72 opt]# /etc/init.d/nginx start Starting nginx daemon: nginxngx_http_fastdfs_set pid=21702#查看jemalloc內存管理
lsof -n | grep jemalloc###—————— tracker server端做nginx反向代理 ———————-###
#nginx.conf配置
upstream fdfs_group1 { server 192.168.5.72 weight=1 max_fails=2 fail_timeout=30s; server 192.168.5.73 weight=1 max_fails=2 fail_timeout=30s; } server { listen 80; server_name 192.168.5.71; location /group1/M00 { proxy_pass http://fdfs_group1; } }###—————————- PHP客戶端配置 ——————————###
#首先客戶端要安裝LNMP環境或者LANMP環境
#因為php的客戶端安裝也會依賴fastdfs本身的一些庫文件，所以要安裝fastdfs
cd /opt/fastdfs/php_client /usr/local/php/bin/phpize ./configure --with-php-config=/usr/local/php/bin/php-config make;make installcat fastdfs_client.ini >> /usr/local/php/etc/php.ini#配置fastDFS的client.conf
mkdir -p /data/fastdfs cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf vi /etc/fdfs/client.conf base_path=/data/fastfds tracker_server=192.168.5.71:22122 http.tracker_server_port=80#重新加載php
service php-fpm reload#驗證模塊是否被正常加載
/usr/local/php/bin/php -m | grep fastdfs_client
#通過http上傳測試
cat test.php
<html> <body> <form action="upload.php" method="post" enctype="multipart/form-data"> <label for="file">Filename:</label> <input type="file" name="upFile" id="upFile" /> <br /> <input type="submit" name="submit" value="Submit" /> </form> </body> </html>cat upload.php
<?php header('Content-type:text/html;charset=utf-8'); //上傳附件 function uploadAttach() {/*{{{*/ $ret = array(); $ret['errorcode'] = 0; $ret['errormsg'] = ''; if(!$_FILES || false == isset($_FILES["upFile"])) { $ret['errorcode'] = 1; $ret['errormsg'] = "ERROR:upFile is not set"; return $ret; } $file = $_FILES["upFile"]; if (false == isset($file['tmp_name']) || false == is_file($file['tmp_name'])) { $ret['errorcode'] = 2; $ret['errormsg'] = "tmp_name is not file"; return $ret; } if (0 == filesize($file['tmp_name'])) { $ret['errorcode'] = 3; $ret['errormsg'] = "tmp_name filesize is 0"; return $ret; } $curlFile = new CurlFile($file['tmp_name'], $file['type'], $file['name']); $fileSuffix = getSuffix($curlFile->getPostFilename()); $ret['file'] = $file; $ret['fileId'] = uploadToFastdfs($curlFile, $fileSuffix); return $ret; }/*}}}*/ //獲取后綴 function getSuffix($fileName) {/*{{{*/ preg_match('/.(w+)?$/', $fileName, $matchs); return isset($matchs[1])?$matchs[1]:''; }/*}}}*/ //上傳文件到fastdfs function uploadToFastdfs(CurlFile $file, $fileSuffix) {/*{{{*/ $fdfs = new FastDFS(); $tracker = $fdfs->tracker_get_connection(); $fileId = $fdfs->storage_upload_by_filebuff1(file_get_contents($file->getFilename()), $fileSuffix); $fdfs->tracker_close_all_connections(); return $fileId; }/*}}}*/ function start() { $ret = uploadAttach(); print_r($ret); } start(); ?>
#通過php腳本上傳測試
echo "php client is here." > /opt/upload.txtcat test.php
<?php var_dump(function_exists('fastdfs_storage_upload_by_filename')); $ret = fastdfs_storage_upload_by_filename('/opt/upload.txt'); var_dump($ret); ?>#執行php腳本
/usr/local/php/bin/php test.php###————————- storage server宕機測試 ————————###
#先關閉storage73
#通過tracker可以看到storage73已經不在線了
#然后通過客戶端上傳一張圖片
#上傳的圖片可以正常訪問
#重啟storage73看上傳的圖片有沒有同步過來
###————————— 橫向擴容，增加group2 ————————###
#添加一組storage server: 192.168.5.74 192.168.5.75（group2）
#分別安裝fastdfs、nginx、fastdfs-nginx-module
#修改group2成員的storage.conf（192.168.5.74 192.168.5.75）
cp /etc/fdfs/storage.conf.sample /etc/fdfs/storage.conf#修改storage server的配置文件：
vi /etc/fdfs/storage.conf group_name=group2 bind_addr=192.168.5.74 base_path=/data/fastdfs store_path0=/data/fastdfs tracker_server=192.168.5.71:22122 http.server_port=80
#創建數據目錄，并啟動storage server
mkdir -p /data/fastdfs /etc/init.d/fdfs_storaged start#開機啟動
/sbin/chkconfig --add fdfs_storaged /sbin/chkconfig --level 2345 fdfs_storaged on#修改所有storage server的mod_fastdfs.conf配置文件
vi /etc/fdfs/mod_fastdfs.conf base_path=/data/fastdfs tracker_server=192.168.5.71:22122 store_path0=/data/fastdfs url_have_group_name = true group_name=group1/group2 # 所有組名 group_count = 2 # 組的總數 # 把所有組和目錄都添加上 [group1] group_name=group1 storage_server_port=23000 store_path_count=1 store_path0=/data/fastdfs [group2] group_name=group2 storage_server_port=23000 store_path_count=1 store_path0=/data/fastdfs
#在所有storage機器上，創建所有組的store_path目錄
mkdir -p /data/fastdfs#修改所有storage server的nginx配置文件
server { listen 80; server_name localhost; location ~/group([0-9])/M00/{ ngx_fastdfs_module; } }#重啟storage server的nginx
/etc/init.d/nginx restart#修改tracker server的反向代理
user www www; worker_processes 8; error_log /usr/local/nginx/logs/nginx_error.log crit; pid /usr/local/nginx/logs/nginx.pid; worker_rlimit_nofile 65535; #工作模式及連接數上限 events { worker_connections 65535; } http { include mime.types; default_type application/octet-stream; #log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"'; #access_log logs/access.log main; sendfile off; tcp_nopush on; keepalive_timeout 300; #nginx跟后端服務器連接超時時間(代理連接超時) proxy_connect_timeout 300s; #連接成功后，后端服務器響應時間(代理接收超時) proxy_read_timeout 300s; proxy_send_timeout 300s; #設置代理服務器（nginx）保存用戶頭信息的緩沖區大小 proxy_buffer_size 64k; #proxy_buffers緩沖區，網頁平均在32k以下的話，這樣設置 proxy_buffers 4 32k; #高負荷下緩沖大小（proxy_buffers*2） proxy_busy_buffers_size 64k; #設定緩存文件夾大小，大于這個值，將從upstream服務器傳遞請求，而不緩沖到磁盤 proxy_temp_file_write_size 64k; #不允許代理端主動關閉連接 proxy_ignore_client_abort on; proxy_cache_path /data/cache1 levels=1:2 keys_zone=fastdfs_cache:200m inactive=1d max_size=20g; fastcgi_connect_timeout 300; fastcgi_send_timeout 300; fastcgi_read_timeout 300; fastcgi_buffer_size 128k; fastcgi_buffers 4 128k; fastcgi_busy_buffers_size 128k; fastcgi_temp_file_write_size 128k; gzip on; gzip_min_length 1k; gzip_buffers 4 16k; gzip_http_version 1.0; gzip_comp_level 2; gzip_types text/plain application/x-javascript text/css application/xml; gzip_vary on; #nginx cache命中率統計日志 log_format cachelog '$remote_addr - $remote_user [$time_local] "$request" ' '$status "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for" ' '"$upstream_cache_status" $body_bytes_sent'; #設置 group1 的服務器 upstream fdfs_group1 { server 192.168.5.72 weight=1 max_fails=2 fail_timeout=30s; server 192.168.5.73 weight=1 max_fails=2 fail_timeout=30s; } #設置 group2 的服務器 upstream fdfs_group2 { server 192.168.5.74 weight=1 max_fails=2 fail_timeout=30s; server 192.168.5.75 weight=1 max_fails=2 fail_timeout=30s; } server { listen 80; server_name localhost; #charset koi8-r; #access_log logs/host.access.log main; #設置 group 的負載均衡參數 location /group1/M00 { proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache fastdfs_cache; proxy_cache_valid 200 304 12h; proxy_cache_key $uri$is_args$args; proxy_pass http://fdfs_group1; expires 30d; } location /group2/M00 { proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache fastdfs_cache; proxy_cache_valid 200 304 12h; proxy_cache_key $uri$is_args$args; proxy_pass http://fdfs_group2; expires 30d; } #設置清除緩存的訪問權限 location ~/purge(/.*) { allow 127.0.0.1; allow 192.168.5.0/24; deny all; proxy_cache_purge http-cache $1$is_args$args; } #error_page 404 /404.html; # redirect server error pages to the static page /50x.html # error_page 500 502 503 504 /50x.html; location = /50x.html { root html; } } }#fastdfs狀態監控
[root@storage71 /]# fdfs_monitor /etc/fdfs/client.conf server_count=1, server_index=0 tracker server is 192.168.5.71:22122 group count: 2 Group 1: group name = group1 Storage 1: id = 192.168.5.72 ip_addr = 192.168.5.72 (storage72.blufly.com) ACTIV Storage 2: id = 192.168.5.73 ip_addr = 192.168.5.73 (storage73.blufly.com) WAIT_SYNC Group 2: group name = group2 Storage 1: id = 192.168.5.74 ip_addr = 192.168.5.74 (storage74.blufly.com) ACTIVE Storage 2: id = 192.168.5.75 ip_addr = 192.168.5.75 (storage75.blufly.com) WAIT_SYNC#上傳圖片測試
#fastdfs的storage server的狀態查詢
#FDFS_STORAGE_STATUS：INIT:初始化，尚未得到同步已有數據的源服務器 #FDFS_STORAGE_STATUS：WAIT_SYNC:等待同步，已得到同步已有數據的源服務器 #FDFS_STORAGE_STATUS：SYNCING:同步中 #FDFS_STORAGE_STATUS：DELETED:已刪除，該服務器從本組中摘除 #FDFS_STORAGE_STATUS：OFFLINE:離線 #FDFS_STORAGE_STATUS：ONLINE:在線，尚不能提供服務 #FDFS_STORAGE_STATUS：ACTIVE:在線，可以提供服務