FastDFS
与MinIO
以及云服务(七牛云、阿里云等的对象存储),鉴于FastDFS
配置较为复杂,最终决定使用MinIO
,易上手,可扩展。MinIO
是全球领先的对象存储先锋,在标准硬件上,读/写速度上高达183 GB / 秒 和 171 GB / 秒。MinIO用作云原生应用程序的主要存储,与传统对象存储相比,云原生应用程序需要更高的吞吐量和更低的延迟。通过添加更多集群可以扩展名称空间,更多机架,直到实现目标。同时,符合一切原生云计算的架构和构建过程,并且包含最新的云计算的全新的技术和概念。
关于对象存储,使用起来无非就是文件上传、下载与删除,再加上桶的操作而已。
所以主要功能如下:
官方文档:http://docs.minio.org.cn/docs/master/minio-docker-quickstart-guide
因为docker方便 所以我们先安装docker
执行以下命令
sudo yum install -y yum-utils device-mapper-persistent-data lvm2
yum list docker-ce --showduplicates | sort -r
yum -y install docker-ce-18.03.1.ce
yum -y install docker-ce
systemctl start docker
systemctl enable docker
docker -v
image-20220812225822854
docker run -p 9000:9000 --name minio1 \
-v /mnt/data:/data \
-v /mnt/config:/root/.minio \
minio/minio server /data
image-20220812225925623
image-20220812230040068
发现ctrl+c后 容器就停止了
image-20220812230112085
原因是我们的docker是运行在前台的 我们需要使用-d 或者ctrl + p + q退出 (使用-d比较好)
正确命令
docker run -p 9000:9000 -p 9090:9090 \
--net=host \
--name minio \
-d --restart=always \
-e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" \
-e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \
-v /mydata/minio/data:/mydata/minio/data \
-v /mydata/minio/config:/mydata/minio/config \
minio/minio server \
/data --console-address ":9090" -address ":9000"
这里的 \ 指的是命令还没有输入完,还需要继续输入命令,先不要执行的意思。这里的9090端口指的是minio的客户端端口。虽然设置9090,但是我们在访问9000的时候,他会自动跳到9090。
image-20220812231406628
这时候运行成功 我们去添加反向代理 进入管理端
image-20220812231645818
image-20220812231804654
到这里我们的安装就结束了
其实这个bucketName
就是文件夹的意思,我们要把文件上传到哪个bucketName
,就是要把文件上传到对应的目录下。
image-20220812232006251
<!-- https://mvnrepository.com/artifact/io.minio/minio -->
<!-- https://mvnrepository.com/artifact/io.minio/minio -->
<dependency>
<groupId>io.minio</groupId>
<artifactId>minio</artifactId>
<version>8.4.3</version>
</dependency>
<!--web 模块-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<!-- hutool工具类 -->
<dependency>
<groupId>cn.hutool</groupId>
<artifactId>hutool-all</artifactId>
<version>5.8.5</version>
</dependency>
<!-- 工具类依赖 -->
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
<version>3.12.0</version>
</dependency>
<dependency>
<groupId>commons-fileupload</groupId>
<artifactId>commons-fileupload</artifactId>
<version>1.4</version>
</dependency>
<!-- 压缩图片 -->
<dependency>
<groupId>net.coobird</groupId>
<artifactId>thumbnailator</artifactId>
<version>0.4.17</version>
</dependency>
package cn.jxwazx.oss;
import io.minio.MinioClient;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
@Configuration
@EnableConfigurationProperties(MinioProperties.class)
public class MinioConfig {
@Autowired
private MinioProperties minioProperties;
@Bean
public MinioClient minioClient(){
return MinioClient.builder()
.endpoint(minioProperties.getEndpoint())
.credentials(minioProperties.getAccessKey(),minioProperties.getSecretKey())
.build();
}
}
package cn.jxwazx.oss;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.stereotype.Component;
@ConfigurationProperties(prefix = "minio")
@Component
@Data
public class MinioProperties {
/**
* 连接地址
*/
private String endpoint;
/**
* 用户名
*/
private String accessKey;
/**
* 密码
*/
private String secretKey;
/**
* 域名
*/
private String nginxHost;
}
package cn.jxwazx.oss;
import cn.hutool.core.date.DateUtil;
import io.minio.*;
import io.minio.errors.*;
import io.minio.messages.Bucket;
import lombok.extern.slf4j.Slf4j;
import net.coobird.thumbnailator.Thumbnails;
import org.apache.commons.fileupload.FileItem;
import org.apache.commons.fileupload.FileItemFactory;
import org.apache.commons.fileupload.disk.DiskFileItemFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.web.multipart.MultipartFile;
import org.springframework.web.multipart.commons.CommonsMultipartFile;
import java.io.File;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.security.InvalidKeyException;
import java.security.NoSuchAlgorithmException;
import java.util.List;
import java.util.Optional;
import java.util.Random;
@Component
@Slf4j
public class MinioUtil {
@Autowired
private MinioProperties minioProperties;
@Autowired
private MinioClient minioClient;
private final Long maxSize = (long) (1024 * 1024);
/**
* 创建bucket
*/
public void createBucket(String bucketName) throws Exception {
if (!minioClient.bucketExists(BucketExistsArgs.builder().bucket(bucketName).build())) {
minioClient.makeBucket(MakeBucketArgs.builder().bucket(bucketName).build());
}
}
/**
* 上传文件
*/
public UploadResponse uploadFile(MultipartFile file, String bucketName) throws Exception {
//判断文件是否为空
if (null == file || 0 == file.getSize()) {
return null;
}
//判断存储桶是否存在 不存在则创建
createBucket(bucketName);
//文件名
String originalFilename = file.getOriginalFilename();
//新的文件名 = 时间戳_随机数.后缀名
assert originalFilename != null;
long now = System.currentTimeMillis() / 1000;
String fileName = DateUtil.format(DateUtil.date(),"yyyyMMdd")+"_"+ now + "_" + new Random().nextInt(1000) +
originalFilename.substring(originalFilename.lastIndexOf("."));
//开始上传
log.info("file压缩前大小:{}",file.getSize());
if (file.getSize() > maxSize) {
FileItemFactory fileItemFactory = new DiskFileItemFactory();
FileItem fileItem = fileItemFactory.createItem(fileName, "text/plain", true, fileName);
OutputStream outputStream = fileItem.getOutputStream();
Thumbnails.of(file.getInputStream()).scale(1f).outputFormat(originalFilename.substring(originalFilename.lastIndexOf(".")+1)).outputQuality(0.25f).toOutputStream(outputStream);
file = new CommonsMultipartFile(fileItem);
}
log.info("file压缩后大小:{}",file.getSize());
minioClient.putObject(
PutObjectArgs.builder().bucket(bucketName).object(fileName).stream(
file.getInputStream(), file.getSize(), -1)
.contentType(file.getContentType())
.build());
String url = minioProperties.getEndpoint() + "/" + bucketName + "/" + fileName;
String urlHost = minioProperties.getNginxHost() + "/" + bucketName + "/" + fileName;
return new UploadResponse(url, urlHost);
}
/**
* 获取全部bucket
*
* @return
*/
public List<Bucket> getAllBuckets() throws Exception {
return minioClient.listBuckets();
}
/**
* 根据bucketName获取信息
*
* @param bucketName bucket名称
*/
public Optional<Bucket> getBucket(String bucketName) throws IOException, InvalidKeyException, NoSuchAlgorithmException, InsufficientDataException, InvalidResponseException, InternalException, ErrorResponseException, ServerException, XmlParserException, ServerException {
return minioClient.listBuckets().stream().filter(b -> b.name().equals(bucketName)).findFirst();
}
/**
* 根据bucketName删除信息
*
* @param bucketName bucket名称
*/
public void removeBucket(String bucketName) throws Exception {
minioClient.removeBucket(RemoveBucketArgs.builder().bucket(bucketName).build());
}
/**
* 获取⽂件外链
*
* @param bucketName bucket名称
* @param objectName ⽂件名称
* @param expires 过期时间 <=7
* @return url
*/
public String getObjectURL(String bucketName, String objectName, Integer expires) throws Exception {
return minioClient.getPresignedObjectUrl(GetPresignedObjectUrlArgs.builder().bucket(bucketName).object(objectName).expiry(expires).build());
}
/**
* 获取⽂件
*
* @param bucketName bucket名称
* @param objectName ⽂件名称
* @return ⼆进制流
*/
public InputStream getObject(String bucketName, String objectName) throws Exception {
return minioClient.getObject(GetObjectArgs.builder().bucket(bucketName).object(objectName).build());
}
/**
* 上传⽂件
*
* @param bucketName bucket名称
* @param objectName ⽂件名称
* @param stream ⽂件流
* @throws Exception https://docs.minio.io/cn/java-minioClient-api-reference.html#putObject
*/
public void putObject(String bucketName, String objectName, InputStream stream) throws
Exception {
minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object(objectName).stream(stream, stream.available(), -1).contentType(objectName.substring(objectName.lastIndexOf("."))).build());
}
/**
* 上传⽂件
*
* @param bucketName bucket名称
* @param objectName ⽂件名称
* @param stream ⽂件流
* @param size ⼤⼩
* @param contextType 类型
* @throws Exception https://docs.minio.io/cn/java-minioClient-api-reference.html#putObject
*/
public void putObject(String bucketName, String objectName, InputStream stream, long
size, String contextType) throws Exception {
minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object(objectName).stream(stream, size, -1).contentType(contextType).build());
}
/**
* 获取⽂件信息
*
* @param bucketName bucket名称
* @param objectName ⽂件名称
* @throws Exception https://docs.minio.io/cn/java-minioClient-api-reference.html#statObject
*/
public StatObjectResponse getObjectInfo(String bucketName, String objectName) throws Exception {
return minioClient.statObject(StatObjectArgs.builder().bucket(bucketName).object(objectName).build());
}
/**
* 删除⽂件
*
* @param bucketName bucket名称
* @param objectName ⽂件名称
* @throws Exception https://docs.minio.io/cn/java-minioClient-apireference.html#removeObject
*/
public void removeObject(String bucketName, String objectName) throws Exception {
minioClient.removeObject(RemoveObjectArgs.builder().bucket(bucketName).object(objectName).build());
}
/***
* 上传视频
* @param file
* @param bucketName
* @return
* @throws Exception
*/
public UploadResponse uploadVideo(MultipartFile file, String bucketName) throws Exception {
//判断文件是否为空
if (null == file || 0 == file.getSize()) {
return null;
}
//判断存储桶是否存在 不存在则创建
createBucket(bucketName);
//文件名
String originalFilename = file.getOriginalFilename();
//新的文件名 = 时间戳_随机数.后缀名
assert originalFilename != null;
long now = System.currentTimeMillis() / 1000;
String fileName = DateUtil.format(DateUtil.date(),"yyyyMMdd")+"_"+ now + "_" + new Random().nextInt(1000) +
originalFilename.substring(originalFilename.lastIndexOf("."));
//开始上传
log.info("file大小:{}",file.getSize());
minioClient.putObject(
PutObjectArgs.builder().bucket(bucketName).object(fileName).stream(
file.getInputStream(), file.getSize(), -1)
.contentType("video/mp4")
.build());
String url = minioProperties.getEndpoint() + "/" + bucketName + "/" + fileName;
String urlHost = minioProperties.getNginxHost() + "/" + bucketName + "/" + fileName;
return new UploadResponse(url, urlHost);
}
}
package cn.jxwazx.oss;
import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;
@Data
@NoArgsConstructor
@AllArgsConstructor
public class UploadResponse {
private String minIoUrl;
private String nginxUrl;
}
package cn.jxwazx.oss;
import cn.jxwazx.smart.entity.R;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.multipart.MultipartFile;
@RestController
@Slf4j
public class TestController {
@Autowired
private MinioUtil minioUtil;
/**
* @author: xx
* @date: 2022/5/25 15:32
* @description: 上传文件
*/
@PostMapping("/upload")
public R minioUpload(@RequestParam(value = "file") MultipartFile file) {
UploadResponse response = null;
try {
response = minioUtil.uploadFile(file, "bucket01");
} catch (Exception e) {
log.error("上传失败", e);
}
return R.success(response);
}
/**
* @author: xx
* @date: 2022/5/25 15:32
* @description: 上传视频
*/
@PostMapping("/uploadVideo")
public R uploadVideo(@RequestParam(value = "file") MultipartFile file) {
UploadResponse response = null;
try {
response = minioUtil.uploadVideo(file, "video-test");
} catch (Exception e) {
log.error("上传失败", e);
}
return R.success(response);
}
}
image-20220813002901863
访问报错
image-20220813002925310
原因:bucket默认是私有的 这是权限问题 我们需要手动开启
image-20220813003225520
image-20220813003545426
原因:我们工具类中写了是text/plain()
image-20220813003732079
package cn.jxwazx.oss;
import org.apache.commons.lang3.StringUtils;
import lombok.Getter;
/**
* 依据文件后缀名返回ContentType
* @author zzg
*
*/
@Getter
public enum MimeTypeEnum {
AAC("acc", "AAC音频", "audio/aac"),
ABW("abw", "AbiWord文件", "application/x-abiword"),
ARC("arc", "存档文件", "application/x-freearc"),
AVI("avi", "音频视频交错格式", "video/x-msvideo"),
AZW("azw", "亚马逊Kindle电子书格式", "application/vnd.amazon.ebook"),
BIN("bin", "任何类型的二进制数据", "application/octet-stream"),
BMP("bmp", "Windows OS / 2位图图形", "image/bmp"),
BZ("bz", "BZip存档", "application/x-bzip"),
BZ2("bz2", "BZip2存档", "application/x-bzip2"),
CSH("csh", "C-Shell脚本", "application/x-csh"),
CSS("css", "级联样式表(CSS)", "text/css"),
CSV("csv", "逗号分隔值(CSV)", "text/csv"),
DOC("doc", "微软Word文件", "application/msword"),
DOCX("docx", "Microsoft Word(OpenXML)", "application/vnd.openxmlformats-officedocument.wordprocessingml.document"),
EOT("eot", "MS Embedded OpenType字体", "application/vnd.ms-fontobject"),
EPUB("epub", "电子出版物(EPUB)", "application/epub+zip"),
GZ("gz", "GZip压缩档案", "application/gzip"),
GIF("gif", "图形交换格式(GIF)", "image/gif"),
HTM("htm", "超文本标记语言(HTML)", "text/html"),
HTML("html", "超文本标记语言(HTML)", "text/html"),
ICO("ico", "图标格式", "image/vnd.microsoft.icon"),
ICS("ics", "iCalendar格式", "text/calendar"),
JAR("jar", "Java存档", "application/java-archive"),
JPEG("jpeg", "JPEG图像", "image/jpeg"),
JPG("jpg", "JPEG图像", "image/jpeg"),
JS("js", "JavaScript", "text/javascript"),
JSON("json", "JSON格式", "application/json"),
JSONLD("jsonld", "JSON-LD格式", "application/ld+json"),
MID("mid", "乐器数字接口(MIDI)", "audio/midi"),
MIDI("midi", "乐器数字接口(MIDI)", "audio/midi"),
MJS("mjs", "JavaScript模块", "text/javascript"),
MP3("mp3", "MP3音频", "audio/mpeg"),
MPEG("mpeg", "MPEG视频", "video/mpeg"),
MPKG("mpkg", "苹果安装程序包", "application/vnd.apple.installer+xml"),
ODP("odp", "OpenDocument演示文稿文档", "application/vnd.oasis.opendocument.presentation"),
ODS("ods", "OpenDocument电子表格文档", "application/vnd.oasis.opendocument.spreadsheet"),
ODT("odt", "OpenDocument文字文件", "application/vnd.oasis.opendocument.text"),
OGA("oga", "OGG音讯", "audio/ogg"),
OGV("ogv", "OGG视频", "video/ogg"),
OGX("ogx", "OGG", "application/ogg"),
OPUS("opus", "OPUS音频", "audio/opus"),
OTF("otf", "otf字体", "font/otf"),
PNG("png", "便携式网络图形", "image/png"),
PDF("pdf", "Adobe 可移植文档格式(PDF)", "application/pdf"),
PHP("php", "php", "application/x-httpd-php"),
PPT("ppt", "Microsoft PowerPoint", "application/vnd.ms-powerpoint"),
PPTX("pptx", "Microsoft PowerPoint(OpenXML)", "application/vnd.openxmlformats-officedocument.presentationml.presentation"),
RAR("rar", "RAR档案", "application/vnd.rar"),
RTF("rtf", "富文本格式", "application/rtf"),
SH("sh", "Bourne Shell脚本", "application/x-sh"),
SVG("svg", "可缩放矢量图形(SVG)", "image/svg+xml"),
SWF("swf", "小型Web格式(SWF)或Adobe Flash文档", "application/x-shockwave-flash"),
TAR("tar", "磁带存档(TAR)", "application/x-tar"),
TIF("tif", "标记图像文件格式(TIFF)", "image/tiff"),
TIFF("tiff", "标记图像文件格式(TIFF)", "image/tiff"),
TS("ts", "MPEG传输流", "video/mp2t"),
TTF("ttf", "ttf字体", "font/ttf"),
TXT("txt", "文本(通常为ASCII或ISO 8859- n", "text/plain"),
VSD("vsd", "微软Visio", "application/vnd.visio"),
WAV("wav", "波形音频格式", "audio/wav"),
WEBA("weba", "WEBM音频", "audio/webm"),
WEBM("webm", "WEBM视频", "video/webm"),
WEBP("webp", "WEBP图像", "image/webp"),
WOFF("woff", "Web开放字体格式(WOFF)", "font/woff"),
WOFF2("woff2", "Web开放字体格式(WOFF)", "font/woff2"),
XHTML("xhtml", "XHTML", "application/xhtml+xml"),
XLS("xls", "微软Excel", "application/vnd.ms-excel"),
XLSX("xlsx", "微软Excel(OpenXML)", "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"),
XML("xml", "XML", "application/xml"),
XUL("xul", "XUL", "application/vnd.mozilla.xul+xml"),
ZIP("zip", "ZIP", "application/zip"),
MIME_3GP("3gp", "3GPP audio/video container", "video/3gpp"),
MIME_3GP_WITHOUT_VIDEO("3gp", "3GPP audio/video container doesn't contain video", "audio/3gpp2"),
MIME_3G2("3g2", "3GPP2 audio/video container", "video/3gpp2"),
MIME_3G2_WITHOUT_VIDEO("3g2", "3GPP2 audio/video container doesn't contain video", "audio/3gpp2"),
MIME_7Z("7z", "7-zip存档", "application/x-7z-compressed");
//扩展名
private final String extension;
//说明
private final String explain;
//contentType/mime类型
private final String mimeType;
/**
* @param extension 上传的文件扩展名
* @param explain 类型说明
* @param mimeType Mime对应的类型
*/
MimeTypeEnum(String extension, String explain, String mimeType) {
this.extension = extension;
this.explain = explain;
this.mimeType = mimeType;
}
/**
* 通过扩展名获取枚举类型
*
* @param extension 扩展名
* @return 枚举类
*/
public static MimeTypeEnum getByExtension(String extension) {
if (StringUtils.isEmpty(extension)) {
return null;
}
for (MimeTypeEnum typesEnum : MimeTypeEnum.values()) {
if (extension.equalsIgnoreCase(typesEnum.getExtension())) {
return typesEnum;
}
}
return null;
}
/**
* Content-Type常用对照
* 根据后缀获取Mime
*
* @param fileType 扩展名
* @return mime类型
*/
public static String getContentType(String fileType) {
MimeTypeEnum mimeTypeEnum = MimeTypeEnum.getByExtension(fileType);
if (mimeTypeEnum != null) {
return mimeTypeEnum.getMimeType();
}
return "application/octet-stream";
}
}
/**
* 上传文件
*/
public UploadResponse uploadFile(MultipartFile file, String bucketName) throws Exception {
//判断文件是否为空
if (null == file || 0 == file.getSize()) {
return null;
}
//判断存储桶是否存在 不存在则创建
createBucket(bucketName);
//文件名
String originalFilename = file.getOriginalFilename();
//新的文件名 = 时间戳_随机数.后缀名
assert originalFilename != null;
long now = System.currentTimeMillis() / 1000;
String fileName = DateUtil.format(DateUtil.date(),"yyyyMMdd")+"_"+ now + "_" + new Random().nextInt(1000) +
originalFilename.substring(originalFilename.lastIndexOf("."));
//开始上传
log.info("file压缩前大小:{}",file.getSize());
if (file.getSize() > maxSize) {
FileItemFactory fileItemFactory = new DiskFileItemFactory();
FileItem fileItem = fileItemFactory.createItem(fileName,
//其实只改了这里 MimeTypeEnum.getContentType(originalFilename.substring(originalFilename.lastIndexOf(".") +1)), true, fileName);
OutputStream outputStream = fileItem.getOutputStream();
Thumbnails.of(file.getInputStream()).scale(1f).outputFormat(originalFilename.substring(originalFilename.lastIndexOf(".")+1)).outputQuality(0.25f).toOutputStream(outputStream);
file = new CommonsMultipartFile(fileItem);
}
log.info("file压缩后大小:{}",file.getSize());
minioClient.putObject(
PutObjectArgs.builder().bucket(bucketName).object(fileName).stream(
file.getInputStream(), file.getSize(), -1)
.contentType(file.getContentType())
.build());
String url = minioProperties.getEndpoint() + "/" + bucketName + "/" + fileName;
String urlHost = minioProperties.getNginxHost() + "/" + bucketName + "/" + fileName;
return new UploadResponse(url, urlHost);
}
image-20220813004955478
参考文章:https://juejin.cn/post/7101581935486615559#heading-17