事前准备
1. ntp时间必须同步。尤其虚拟机测试(经常踩坑)
2.分布式Minio里所有的节点需要有同样的access秘钥和secret秘钥,这样这些节点才能建立联接。为了实现这个,你需要在执行minio server命令之前,先将access秘钥和secret秘钥export成环境变量。
3.分布式Minio使用的磁盘里必须是干净的,里面没有数据。
4.下面示例里的IP仅供示例参考,你需要改成你真实用到的IP和文件夹路径。 如:192.168.1.1/data (真实IP + 真实目录)
测试环境
五台 测试虚拟机(centos7):m1 m2 m3 m4 m5 m1: nginx (官方全列建议用负载均衡管控) m2-5: MinIO server
1. 同步配置host
# cat /etc/hosts 192.168.77.101 k8s1 minio.go 192.168.77.104 k8s4 192.168.77.105 k8s5 192.168.77.106 k8s6 192.168.77.107 k8s7
2. 部署minio server
wget https://dl.min.io/server/minio/release/linux-amd64/minio mv minio /usr/local/bin/minio chmod +x /usr/local/bin/minio
3. 部署启动守护进程
# cat /lib/systemd/system/minio.service [Unit] Description=MinIO Documentation=https://docs.min.io Wants=network-online.target After=network-online.target AssertFileIsExecutable=/usr/local/bin/minio [Service] WorkingDirectory=/usr/local #备注 user和group 官方用的是minio-user 如果用三方用户 需要给目录授权 不然回报错,测试服务,所以改成了root User=root Group=root #ProtectProc=invisible #这个是一个官方BUG EnvironmentFile=-/etc/default/minio #启动的env 环境变量调用 ExecStartPre=/bin/bash -c "if [ -z \"${MINIO_VOLUMES}\" ]; then echo \"Variable MINIO_VOLUMES not set in /etc/default/minio\"; exit 1; fi" ExecStart=/usr/local/bin/minio server $MINIO_OPTS $MINIO_VOLUMES # Let systemd restart this service always Restart=always # Specifies the maximum file descriptor number that can be opened by this process LimitNOFILE=65535 # Specifies the maximum number of threads this process can create TasksMax=infinity # Disable timeout logic and wait until process is stopped TimeoutStopSec=infinity SendSIGKILL=no [Install] WantedBy=multi-user.target # Built for ${project.name}-${project.version} (${project.name}) 由于没有env目录 需要创建一个 # mkdir /etc/default/ 配置环境变量 # cat /etc/default/minio MINIO_VOLUMES="http://k8s{4...7}:9000/export" # Set all MinIO server options # The following explicitly sets the MinIO Console listen address to # port 9001 on all network interfaces. The default behavior is dynamic # port selection. MINIO_OPTS="--console-address :9001" # Defer to your organizations requirements for superadmin user name. MINIO_ROOT_USER=minio # Set the root password MINIO_ROOT_PASSWORD=password # nginx服务器地址 #MINIO_SERVER_URL="http://minio.go"
4. 创建目录
1. 创建目录必须存在 2. 权限必须正确,和驱动进程一致 3. 此目录,不能为根/ 不能和系统在一起。(如果在一起回报错) 新建一个干净的分区挂载目录 # fdisk -l /dev/sdb Disk /dev/sdb: 10.7 GB, 10737418240 bytes, 20971520 sectors Device Boot Start End Blocks Id System /dev/sdb1 2048 20971519 10484736 83 Linux # df -h /export Filesystem Size Used Avail Use% Mounted on /dev/sdb1 10G 35M 10G 1% /export
5. 启动minio
# systemctl start minio # systemctl status minio ● minio.service - MinIO Loaded: loaded (/usr/lib/systemd/system/minio.service; disabled; vendor preset: disabled) Active: active (running) since Thu 2022-04-18 22:45:02 CST; 20h 7min ago
6. MINIO CLIENT
下载MC,并授权管理 # wget https://dl.min.io/client/mc/release/linux-amd64/mc -O /usr/local/bin/mc # chmod +x /usr/local/bin/mc docker版本 # docker pull minio/mc:latest # docker run -it --entrypoint=/bin/sh minio/mc 简单操作 sh-4.4# mc alias set local http://minio.go:9000 minio password --api s3v4 Added `local` successfully. sh-4.4# mc alias ls local URL : http://192.168.77.107:9000 AccessKey : minio SecretKey : password API : s3v4 Path : auto sh-4.4# mc ls local //--debug [2022-04-21 15:18:17 UTC] 0B stroage/ 创建用户 mc admin policy list minio mc admin policy set minio consoleAdmin user=cn=张三,ou=test-department,ou=NJ-Dev,ou=SFERE-RD,dc=sfere-elec,dc=com mc admin policy set minio readwrite group=cn=jira-software-users,dc=sfere-elec,dc=com mc admin policy set minio consoleAdmin group=cn=超级用户,dc=sfere-elec,dc=com
7. 扩展K8S兼容性
目前来看minio是否满足兼容CSI 作为容器的pv当本地存储来用,目前来看很不成熟。star少的可怜。生产环境不建议上。测试环境
https://github.com/minio/directpv (官推 200多star)
ctrox/csi-s3: A Container Storage Interface for S3 (github.com) (明确表示实验环境,切勿上生产)
参考文档:
Deploy MinIO in Distributed Mode — MinIO Baremetal Documentation
MinIO | Code and downloads to create high performance object storage
没有帐号? 立即注册