I recently learned about Distributed File Systems (DFS) and the benefits they could bring to an organization whose needs require redundant and highly available information across their systems. As part of a class project, I had to look into MooseFS, a fault-tolerant, network based DFS that can be mounted to virtual disks on client machines. Below is a guide on how to set this infrastructure up using CentOS on a practice “how-to” level, rather than a full-scale production.
NOTE: I have not configured MooseFS on a production scale, however modifying the specs within this guide should allow for that capability. I would still recommend checking out the MooseFS official documentation to be sure.
What you need:
The below specs are guidelines for a practice environment and can be set to whatever you need them to be:
- 4 CentOS servers of default storage (16-20GB) placed on your LAN
- 2 CentOS servers with two HDDs (16-20GB & ~15GB) attached and placed on your LAN, keeping one of the drives un-configured during OS installation
Networking:
The following are the hostnames and IP addresses I have used for this guide. Feel free to use whatever you would like, however make sure you document what you choose for easier installation moving forward:
- moosefs-master – 10.0.5.20
- moosefs-meta – 10.0.5.21
- Moosefs-chunk01 (2 HDDs) – 10.0.5.22
- moosefs-chunk02 (2 HDDs) – 10.0.5.23
- linux01 – 10.0.5.91
- linux02 – 10.0.5.92
Brief Architecture Overview
Master Server – the machines managing the whole file system, storing metadata for every file and folder; central point of the cluster.
Chunkservers (Data Servers) – the machines that store files’ data in “chunks”. They serve data for clients and synchronize among themselves when necessary.
Metaloggers (Backup Server) – the machines that store metadata changelogs. They can be easily made a master server in the event of master server failure.
Client Machine – machines that utilize the MooseFS TCP/IP network protocol and mount the filesystem being provided by the MooseFS cluster.
Setup MooseFS Master Server (moosefs-master)
Configure MooseFS Repository
curl "https://ppa.moosefs.com/RPM-GPG-KEY-MooseFS" > /etc/pki/rpm-gpg/RPM-GPG-KEY-MooseFS
curl "http://ppa.moosefs.com/MooseFS-3-el7.repo" > /etc/yum.repos.d/MooseFS.repo
Install MooseFS Master Server and Web GUI
yum install moosefs-master moosefs-cgi moosefs-cgiserv moosefs-cli
Start MooseFS Master Server and Web GUI as a service
systemctl enable moosefs-master
systemctl start moosefs-master
systemctl enable moosefs-cgiserv
systemctl start moosefs-cgiserv
Add Firewall Rules for MooseFS services
firewall-cmd --zone=public --add-service=http --permanent
firewall-cmd --zone=public --add-port=9425/tcp --permanent
firewall-cmd --zone=public --add-port=9419/tcp --permanent
firewall-cmd --zone=public --add-port=9420/tcp --permanent
firewall-cmd --zone=public --add-port=9421/tcp --permanent
firewall-cmd --reload
Access Web GUI via <IPofMasterServer>:9425 in your Browser
Setup MooseFS Metalogger (moosefs-meta)
Configure MooseFS Repository
curl "https://ppa.moosefs.com/RPM-GPG-KEY-MooseFS" > /etc/pki/rpm-gpg/RPM-GPG-KEY-MooseFS
curl "http://ppa.moosefs.com/MooseFS-3-el7.repo" > /etc/yum.repos.d/MooseFS.repo
Install MooseFS Metalogger
yum install moosefs-metalogger
Edit MooseFS Metalogger to point to DNS name of Master Server
cd /etc/mfs
nano mfsmetalogger.cfg (change name of desired master to DNS name)
Start MooseFS Metalogger as a service
systemctl enable moosefs-metalogger
systemctl start moosefs-metalogger
Setup MooseFS Chunkservers (moosefs-chunk01)
Configure MooseFS Repository
curl "https://ppa.moosefs.com/RPM-GPG-KEY-MooseFS" > /etc/pki/rpm-gpg/RPM-GPG-KEY-MooseFS
curl "http://ppa.moosefs.com/MooseFS-3-el7.repo" > /etc/yum.repos.d/MooseFS.repo
Install MooseFS Chunkserver
yum install moosefs-chunkserver
Edit MooseFS Chunkserver to point to DNS name of Master Server
cd /etc/mfs
nano mfschunkserver.cfg (change name of desired master to DNS name)
Create/Format Partition on 2nd HDD for Chunkserver
parted -l
parted --align optimal /dev/sdb
(parted) mklabel gpt
(parted) mkpart mfschunks1 0% 100%
(parted) q
Install Dependency:
yum install xfsprogsls
mkfs.xfs /dev/sdb1
Add the following to /etc/fstab:
/dev/sdb1/ /mnt/mfschunks1 xfs defaults 0 0
Make Mount Directory and Mount New Partition:
mkdir /mnt/mfschunks1
mount /mnt/mfschunks1
Change ownership so MooseFS can make changes:
chown mfs:mfs /mnt/mfschunks1
chmod 770 /mnt/mfschunks1
Append the following to /etc/mfs/mfshdd.cfg:
/mnt/mfschunks1
Start MooseFS Chunkserver as a service
systemctl enable moosefs-chunkserver
systemctl start moosefs-chunkserver
NOTE: Repeat the above steps for ALL additional chunkservers you wish to add. For this guide, we only have moosefs-chunk02 left to do.
Configure Clients (linux01 & linux02)
Configure MooseFS Repository
curl "https://ppa.moosefs.com/RPM-GPG-KEY-MooseFS" > /etc/pki/rpm-gpg/RPM-GPG-KEY-MooseFS
curl "http://ppa.moosefs.com/MooseFS-3-el7.repo" > /etc/yum.repos.d/MooseFS.repo
Install MooseFS Client and Dependencies
yum install fuse fuse-devel
yum install moosefs-client
Mount MooseFS Filesystem to a Created Directory
mkdir -p /mnt/mfs
mfsmount /mnt/mfs -H moosefs-master
NOTE: Repeat for ALL other desired Linux Clients. For this guide, we only have linux02 left to do.
Use MooseFS!
MooseFS should now be completely setup. To test this, we will be adding a test file and looking for it to appear on both linux01 and linux02.
Make a file within the mounted directory on linux01:
cd /mnt/mfs
Add contents to that file:
echo “test” > test1
Check the mounted directory on linux02 to see it replicated:
If you have a replicated file, you are all set!
Burnham Forensics 