Excel大数据异步导出通用解决方案
一个开箱即用的异步导出框架,支持大数据量、图片、进度追踪,可快速集成到任何Spring Boot项目
📋 目录
一、方案概述
1.1 解决的问题
- ✅ 导出大量数据时浏览器超时
- ✅ 同步导出导致页面长时间无响应
- ✅ 包含图片等二进制资源导致内存溢出
- ✅ 无法追踪导出进度
- ✅ 导出失败无法重试
1.2 技术架构
┌──────────────┐ ┌──────────────┐│ 前端应用 │ │ 后端服务 ││ │ ①创建导出任务 │ ││ │ ──────────────────> │ ││ │ ②返回任务ID │ ││ │ <────────────────── │ ││ │ │ ││ 进度展示 │ ③轮询任务状态 │ 异步处理 ││ (2秒/次) │ <─────────────────> │ (线程池) ││ │ │ ││ │ ④任务完成通知 │ ││ │ <────────────────── │ ││ │ ⑤下载文件 │ ││ │ ──────────────────> │ │└──────────────┘ └──────────────┘1.3 核心特性
| 特性 | 说明 |
|---|---|
| 🚀 开箱即用 | 只需实现数据查询接口即可使用 |
| 📊 进度追踪 | 实时显示导出进度和处理数量 |
| 🎯 泛型设计 | 支持任意数据类型的导出 |
| 🖼️ 图片支持 | 内置图片异步下载和超时控制 |
| 💾 分批处理 | 避免内存溢出,支持百万级数据 |
| 🔄 断点续传 | 支持任务重试和失败恢复 |
| 🗂️ 历史记录 | 保存导出历史,随时重新下载 |
| 🔐 权限控制 | 用户只能访问自己的导出文件 |
二、快速开始
2.1 Maven依赖
<dependencies> <!-- Spring Boot Starter --> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-web</artifactId> </dependency>
<!-- MyBatis Plus --> <dependency> <groupId>com.baomidou</groupId> <artifactId>mybatis-plus-boot-starter</artifactId> <version>3.5.3.1</version> </dependency>
<!-- POI for Excel --> <dependency> <groupId>org.apache.poi</groupId> <artifactId>poi-ooxml</artifactId> <version>5.2.3</version> </dependency>
<!-- Hutool工具类 --> <dependency> <groupId>cn.hutool</groupId> <artifactId>hutool-all</artifactId> <version>5.8.18</version> </dependency>
<!-- Lombok --> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <optional>true</optional> </dependency></dependencies>2.2 配置文件
spring: # 异步任务配置 task: execution: pool: core-size: 2 max-size: 5 queue-capacity: 100 keep-alive: 60s
# 导出配置export: # 文件存储路径 file-path: /data/export # 文件保留天数 keep-days: 7 # 单用户最大并发任务数 max-concurrent-per-user: 3 # 全局最大并发任务数 max-concurrent-global: 502.3 数据库脚本
-- 导出任务表CREATE TABLE `export_task` ( `id` varchar(32) NOT NULL COMMENT '任务ID', `task_name` varchar(200) DEFAULT NULL COMMENT '任务名称', `task_type` varchar(50) DEFAULT NULL COMMENT '任务类型', `status` tinyint(4) DEFAULT 0 COMMENT '状态:0-待处理 1-处理中 2-已完成 3-失败 4-已过期', `progress` int(11) DEFAULT 0 COMMENT '进度百分比(0-100)', `query_params` text COMMENT '查询参数(JSON)', `total_count` int(11) DEFAULT NULL COMMENT '总记录数', `processed_count` int(11) DEFAULT 0 COMMENT '已处理数量', `file_path` varchar(500) DEFAULT NULL COMMENT '文件存储路径', `file_name` varchar(200) DEFAULT NULL COMMENT '文件名', `file_size` bigint(20) DEFAULT NULL COMMENT '文件大小(字节)', `error_msg` text COMMENT '错误信息', `create_user` varchar(32) DEFAULT NULL COMMENT '创建人ID', `create_time` datetime DEFAULT CURRENT_TIMESTAMP COMMENT '创建时间', `finish_time` datetime DEFAULT NULL COMMENT '完成时间', PRIMARY KEY (`id`), KEY `idx_create_user` (`create_user`), KEY `idx_status` (`status`), KEY `idx_create_time` (`create_time`)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COMMENT='导出任务表';三、核心代码
3.1 通用数据提供者接口
package com.example.export.core;
import java.util.List;
/** * 导出数据提供者接口 * @param <T> 数据实体类型 */public interface ExportDataProvider<T> {
/** * 查询总记录数 * @param params 查询参数 * @return 总数 */ int queryTotalCount(String params);
/** * 分页查询数据 * @param params 查询参数 * @param offset 偏移量 * @param limit 每页数量 * @return 数据列表 */ List<T> queryData(String params, int offset, int limit);}3.2 Excel行数据转换器接口
package com.example.export.core;
import org.apache.poi.ss.usermodel.Row;import org.apache.poi.ss.usermodel.Workbook;
/** * Excel行数据转换器 * @param <T> 数据实体类型 */public interface ExcelRowConverter<T> {
/** * 获取表头 * @return 表头数组 */ String[] getHeaders();
/** * 设置列宽 * @return 列宽数组(单位:256分之一字符) */ default int[] getColumnWidths() { return new int[0]; // 默认使用自动宽度 }
/** * 将数据转换为Excel行 * @param workbook Excel工作簿 * @param row Excel行对象 * @param data 数据对象 * @param rowIndex 行索引 */ void convertToRow(Workbook workbook, Row row, T data, int rowIndex);}3.3 通用异步导出处理器
package com.example.export.core;
import cn.hutool.core.date.DateUtil;import cn.hutool.core.util.IdUtil;import lombok.extern.slf4j.Slf4j;import org.apache.poi.ss.usermodel.*;import org.apache.poi.xssf.streaming.SXSSFWorkbook;import org.springframework.beans.factory.annotation.Value;import org.springframework.scheduling.annotation.Async;import org.springframework.stereotype.Component;
import javax.annotation.Resource;import java.io.File;import java.io.FileOutputStream;import java.util.Date;import java.util.List;
/** * 通用异步导出处理器 */@Slf4j@Componentpublic class GenericAsyncExportHandler {
@Resource private ExportTaskService exportTaskService;
@Value("${export.file-path:/data/export}") private String exportFilePath;
/** * 执行导出任务 * @param taskId 任务ID * @param taskType 任务类型 * @param dataProvider 数据提供者 * @param rowConverter 行转换器 * @param queryParams 查询参数(JSON格式) * @param <T> 数据类型 */ @Async("exportTaskExecutor") public <T> void executeExport( String taskId, String taskType, ExportDataProvider<T> dataProvider, ExcelRowConverter<T> rowConverter, String queryParams) {
log.info("开始执行导出任务: taskId={}, taskType={}", taskId, taskType);
SXSSFWorkbook workbook = null; FileOutputStream fos = null;
try { // 1. 查询总数 int totalCount = dataProvider.queryTotalCount(queryParams); log.info("查询到数据总数: {}", totalCount);
if (totalCount == 0) { exportTaskService.updateTaskFailed(taskId, "没有符合条件的数据"); return; }
// 2. 更新任务总数 exportTaskService.updateTaskTotal(taskId, totalCount);
// 3. 创建Excel工作簿(使用流式写入,内存中只保留100行) workbook = new SXSSFWorkbook(100); workbook.setCompressTempFiles(true); // 启用临时文件压缩 Sheet sheet = workbook.createSheet("数据");
// 4. 创建样式 CellStyle headerStyle = createHeaderStyle(workbook); CellStyle dataStyle = createDataStyle(workbook);
// 5. 创建表头 createHeaderRow(sheet, rowConverter.getHeaders(), headerStyle);
// 6. 设置列宽 setColumnWidths(sheet, rowConverter.getColumnWidths());
// 7. 分批处理数据 int batchSize = 100; // 每批处理100条 int processedCount = 0; int offset = 0;
while (processedCount < totalCount) { // 分页查询数据 List<T> dataList = dataProvider.queryData(queryParams, offset, batchSize);
if (dataList == null || dataList.isEmpty()) { break; }
// 写入Excel for (T data : dataList) { int rowNum = processedCount + 1; // 跳过表头 Row row = sheet.createRow(rowNum);
// 转换数据到Excel行 rowConverter.convertToRow(workbook, row, data, rowNum);
processedCount++; }
// 更新进度 int progress = (int) ((processedCount * 100.0) / totalCount); exportTaskService.updateTaskProgress(taskId, progress, processedCount); log.info("导出进度: {}/{} ({}%)", processedCount, totalCount, progress);
offset += batchSize; }
// 8. 保存文件 String fileName = generateFileName(taskType); File exportFile = saveWorkbook(workbook, fileName);
log.info("导出完成: file={}, size={}", exportFile.getAbsolutePath(), exportFile.length());
// 9. 更新任务状态为成功 exportTaskService.updateTaskSuccess( taskId, exportFile.getAbsolutePath(), fileName, exportFile.length() );
} catch (Exception e) { log.error("导出任务执行失败: taskId={}", taskId, e); exportTaskService.updateTaskFailed(taskId, "导出失败: " + e.getMessage()); } finally { // 清理资源 closeQuietly(fos); if (workbook != null) { try { workbook.dispose(); // 清理临时文件 workbook.close(); } catch (Exception e) { log.error("关闭工作簿失败", e); } } } }
/** * 创建表头样式 */ private CellStyle createHeaderStyle(Workbook workbook) { CellStyle style = workbook.createCellStyle();
// 背景色 style.setFillForegroundColor(IndexedColors.GREY_25_PERCENT.getIndex()); style.setFillPattern(FillPatternType.SOLID_FOREGROUND);
// 边框 style.setBorderBottom(BorderStyle.THIN); style.setBorderTop(BorderStyle.THIN); style.setBorderLeft(BorderStyle.THIN); style.setBorderRight(BorderStyle.THIN);
// 对齐方式 style.setAlignment(HorizontalAlignment.CENTER); style.setVerticalAlignment(VerticalAlignment.CENTER);
// 字体 Font font = workbook.createFont(); font.setBold(true); font.setFontHeightInPoints((short) 11); style.setFont(font);
return style; }
/** * 创建数据样式 */ private CellStyle createDataStyle(Workbook workbook) { CellStyle style = workbook.createCellStyle();
// 边框 style.setBorderBottom(BorderStyle.THIN); style.setBorderTop(BorderStyle.THIN); style.setBorderLeft(BorderStyle.THIN); style.setBorderRight(BorderStyle.THIN);
// 对齐方式 style.setAlignment(HorizontalAlignment.LEFT); style.setVerticalAlignment(VerticalAlignment.CENTER); style.setWrapText(true); // 自动换行
return style; }
/** * 创建表头行 */ private void createHeaderRow(Sheet sheet, String[] headers, CellStyle headerStyle) { Row headerRow = sheet.createRow(0); headerRow.setHeight((short) 500); // 设置行高
for (int i = 0; i < headers.length; i++) { Cell cell = headerRow.createCell(i); cell.setCellValue(headers[i]); cell.setCellStyle(headerStyle); } }
/** * 设置列宽 */ private void setColumnWidths(Sheet sheet, int[] columnWidths) { if (columnWidths == null || columnWidths.length == 0) { return; }
for (int i = 0; i < columnWidths.length; i++) { sheet.setColumnWidth(i, columnWidths[i]); } }
/** * 生成文件名 */ private String generateFileName(String taskType) { String timestamp = DateUtil.format(new Date(), "yyyyMMdd_HHmmss"); return String.format("%s_%s.xlsx", taskType, timestamp); }
/** * 保存工作簿到文件 */ private File saveWorkbook(SXSSFWorkbook workbook, String fileName) throws Exception { // 创建按日期分组的目录 String dateDir = DateUtil.format(new Date(), "yyyy/MM/dd"); File exportDir = new File(exportFilePath, dateDir);
if (!exportDir.exists()) { exportDir.mkdirs(); }
File exportFile = new File(exportDir, fileName);
try (FileOutputStream fos = new FileOutputStream(exportFile)) { workbook.write(fos); fos.flush(); }
return exportFile; }
/** * 安全关闭流 */ private void closeQuietly(AutoCloseable closeable) { if (closeable != null) { try { closeable.close(); } catch (Exception e) { log.warn("关闭资源失败", e); } } }}3.4 导出任务实体
package com.example.export.entity;
import com.baomidou.mybatisplus.annotation.IdType;import com.baomidou.mybatisplus.annotation.TableId;import com.baomidou.mybatisplus.annotation.TableName;import lombok.Data;
import java.io.Serializable;import java.util.Date;
/** * 导出任务实体 */@Data@TableName("export_task")public class ExportTask implements Serializable {
private static final long serialVersionUID = 1L;
/** * 任务ID */ @TableId(type = IdType.ASSIGN_ID) private String id;
/** * 任务名称 */ private String taskName;
/** * 任务类型 */ private String taskType;
/** * 状态:0-待处理,1-处理中,2-已完成,3-失败,4-已过期 */ private Integer status;
/** * 进度百分比:0-100 */ private Integer progress;
/** * 查询参数(JSON格式) */ private String queryParams;
/** * 总记录数 */ private Integer totalCount;
/** * 已处理数量 */ private Integer processedCount;
/** * 文件路径 */ private String filePath;
/** * 文件名 */ private String fileName;
/** * 文件大小(字节) */ private Long fileSize;
/** * 错误信息 */ private String errorMsg;
/** * 创建人 */ private String createUser;
/** * 创建时间 */ private Date createTime;
/** * 完成时间 */ private Date finishTime;
// 状态常量 public static final int STATUS_PENDING = 0; public static final int STATUS_PROCESSING = 1; public static final int STATUS_COMPLETED = 2; public static final int STATUS_FAILED = 3; public static final int STATUS_EXPIRED = 4;}3.5 导出任务服务
package com.example.export.service;
import com.baomidou.mybatisplus.extension.service.IService;import com.example.export.entity.ExportTask;
/** * 导出任务服务接口 */public interface ExportTaskService extends IService<ExportTask> {
/** * 创建导出任务 */ String createExportTask(String taskName, String taskType, String queryParams, String createUser);
/** * 更新任务总数 */ void updateTaskTotal(String taskId, int totalCount);
/** * 更新任务进度 */ void updateTaskProgress(String taskId, int progress, int processedCount);
/** * 更新任务状态为完成 */ void updateTaskSuccess(String taskId, String filePath, String fileName, long fileSize);
/** * 更新任务状态为失败 */ void updateTaskFailed(String taskId, String errorMsg);
/** * 检查用户并发任务数限制 */ boolean checkConcurrentLimit(String userId, int maxConcurrent);}package com.example.export.service.impl;
import cn.hutool.core.util.IdUtil;import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;import com.baomidou.mybatisplus.extension.service.impl.ServiceImpl;import com.example.export.entity.ExportTask;import com.example.export.mapper.ExportTaskMapper;import com.example.export.service.ExportTaskService;import org.springframework.stereotype.Service;
import java.util.Date;
/** * 导出任务服务实现 */@Servicepublic class ExportTaskServiceImpl extends ServiceImpl<ExportTaskMapper, ExportTask> implements ExportTaskService {
@Override public String createExportTask(String taskName, String taskType, String queryParams, String createUser) { ExportTask task = new ExportTask(); task.setId(IdUtil.fastSimpleUUID()); task.setTaskName(taskName); task.setTaskType(taskType); task.setStatus(ExportTask.STATUS_PENDING); task.setProgress(0); task.setQueryParams(queryParams); task.setProcessedCount(0); task.setCreateUser(createUser); task.setCreateTime(new Date());
this.save(task); return task.getId(); }
@Override public void updateTaskTotal(String taskId, int totalCount) { ExportTask task = new ExportTask(); task.setId(taskId); task.setTotalCount(totalCount); task.setStatus(ExportTask.STATUS_PROCESSING); this.updateById(task); }
@Override public void updateTaskProgress(String taskId, int progress, int processedCount) { ExportTask task = new ExportTask(); task.setId(taskId); task.setStatus(ExportTask.STATUS_PROCESSING); task.setProgress(progress); task.setProcessedCount(processedCount); this.updateById(task); }
@Override public void updateTaskSuccess(String taskId, String filePath, String fileName, long fileSize) { ExportTask task = new ExportTask(); task.setId(taskId); task.setStatus(ExportTask.STATUS_COMPLETED); task.setProgress(100); task.setFilePath(filePath); task.setFileName(fileName); task.setFileSize(fileSize); task.setFinishTime(new Date()); this.updateById(task); }
@Override public void updateTaskFailed(String taskId, String errorMsg) { ExportTask task = new ExportTask(); task.setId(taskId); task.setStatus(ExportTask.STATUS_FAILED); task.setErrorMsg(errorMsg); task.setFinishTime(new Date()); this.updateById(task); }
@Override public boolean checkConcurrentLimit(String userId, int maxConcurrent) { LambdaQueryWrapper<ExportTask> wrapper = new LambdaQueryWrapper<>(); wrapper.eq(ExportTask::getCreateUser, userId) .in(ExportTask::getStatus, ExportTask.STATUS_PENDING, ExportTask.STATUS_PROCESSING);
long count = this.count(wrapper); return count < maxConcurrent; }}3.6 Mapper接口
package com.example.export.mapper;
import com.baomidou.mybatisplus.core.mapper.BaseMapper;import com.example.export.entity.ExportTask;import org.apache.ibatis.annotations.Mapper;
/** * 导出任务Mapper */@Mapperpublic interface ExportTaskMapper extends BaseMapper<ExportTask> {}3.7 线程池配置
package com.example.export.config;
import org.springframework.context.annotation.Bean;import org.springframework.context.annotation.Configuration;import org.springframework.scheduling.annotation.EnableAsync;import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import java.util.concurrent.Executor;import java.util.concurrent.ThreadPoolExecutor;
/** * 异步任务线程池配置 */@Configuration@EnableAsyncpublic class AsyncExportConfig {
/** * 导出任务线程池 */ @Bean("exportTaskExecutor") public Executor exportTaskExecutor() { ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
// 核心线程数 executor.setCorePoolSize(2);
// 最大线程数 executor.setMaxPoolSize(5);
// 队列容量 executor.setQueueCapacity(100);
// 线程空闲时间(秒) executor.setKeepAliveSeconds(60);
// 线程名称前缀 executor.setThreadNamePrefix("export-task-");
// 拒绝策略:由调用线程处理(确保任务不丢失) executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());
// 等待所有任务结束后再关闭线程池 executor.setWaitForTasksToCompleteOnShutdown(true); executor.setAwaitTerminationSeconds(60);
executor.initialize(); return executor; }}3.8 控制器
package com.example.export.controller;
import cn.hutool.json.JSONUtil;import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;import com.baomidou.mybatisplus.extension.plugins.pagination.Page;import com.example.export.entity.ExportTask;import com.example.export.service.ExportTaskService;import lombok.extern.slf4j.Slf4j;import org.springframework.beans.factory.annotation.Value;import org.springframework.core.io.FileSystemResource;import org.springframework.core.io.Resource;import org.springframework.http.HttpHeaders;import org.springframework.http.MediaType;import org.springframework.http.ResponseEntity;import org.springframework.web.bind.annotation.*;
import java.io.File;import java.io.UnsupportedEncodingException;import java.net.URLEncoder;import java.nio.charset.StandardCharsets;import java.util.HashMap;import java.util.Map;
/** * 导出任务控制器 */@Slf4j@RestController@RequestMapping("/api/export")public class ExportTaskController {
@javax.annotation.Resource private ExportTaskService exportTaskService;
@Value("${export.max-concurrent-per-user:3}") private int maxConcurrentPerUser;
/** * 查询任务状态 */ @GetMapping("/task/status/{taskId}") public ResponseEntity<Map<String, Object>> getTaskStatus(@PathVariable String taskId) { ExportTask task = exportTaskService.getById(taskId);
if (task == null) { return ResponseEntity.ok(createErrorResponse("任务不存在")); }
Map<String, Object> data = new HashMap<>(); data.put("taskId", task.getId()); data.put("taskName", task.getTaskName()); data.put("status", task.getStatus()); data.put("progress", task.getProgress()); data.put("totalCount", task.getTotalCount()); data.put("processedCount", task.getProcessedCount()); data.put("fileName", task.getFileName()); data.put("fileSize", task.getFileSize()); data.put("errorMsg", task.getErrorMsg()); data.put("createTime", task.getCreateTime()); data.put("finishTime", task.getFinishTime());
return ResponseEntity.ok(createSuccessResponse(data)); }
/** * 查询任务列表 */ @GetMapping("/task/list") public ResponseEntity<Map<String, Object>> getTaskList( @RequestParam(defaultValue = "1") int current, @RequestParam(defaultValue = "10") int size, @RequestParam(required = false) String createUser) {
Page<ExportTask> page = new Page<>(current, size); LambdaQueryWrapper<ExportTask> wrapper = new LambdaQueryWrapper<>();
if (createUser != null) { wrapper.eq(ExportTask::getCreateUser, createUser); }
wrapper.orderByDesc(ExportTask::getCreateTime);
Page<ExportTask> result = exportTaskService.page(page, wrapper);
Map<String, Object> data = new HashMap<>(); data.put("records", result.getRecords()); data.put("total", result.getTotal()); data.put("current", result.getCurrent()); data.put("size", result.getSize());
return ResponseEntity.ok(createSuccessResponse(data)); }
/** * 下载导出文件 */ @GetMapping("/task/download/{taskId}") public ResponseEntity<Resource> downloadFile(@PathVariable String taskId) { ExportTask task = exportTaskService.getById(taskId);
if (task == null || task.getStatus() != ExportTask.STATUS_COMPLETED) { return ResponseEntity.badRequest().build(); }
File file = new File(task.getFilePath());
if (!file.exists()) { log.error("文件不存在: {}", task.getFilePath()); return ResponseEntity.notFound().build(); }
try { // 文件名编码 String encodedFileName = URLEncoder.encode(task.getFileName(), StandardCharsets.UTF_8.toString()) .replaceAll("\\+", "%20");
Resource resource = new FileSystemResource(file);
return ResponseEntity.ok() .contentType(MediaType.APPLICATION_OCTET_STREAM) .header(HttpHeaders.CONTENT_DISPOSITION, "attachment; filename*=UTF-8''" + encodedFileName) .body(resource);
} catch (UnsupportedEncodingException e) { log.error("文件名编码失败", e); return ResponseEntity.internalServerError().build(); } }
/** * 删除任务 */ @DeleteMapping("/task/{taskId}") public ResponseEntity<Map<String, Object>> deleteTask(@PathVariable String taskId) { ExportTask task = exportTaskService.getById(taskId);
if (task == null) { return ResponseEntity.ok(createErrorResponse("任务不存在")); }
// 删除文件 if (task.getFilePath() != null) { File file = new File(task.getFilePath()); if (file.exists()) { file.delete(); } }
// 删除记录 exportTaskService.removeById(taskId);
return ResponseEntity.ok(createSuccessResponse("删除成功")); }
/** * 创建成功响应 */ private Map<String, Object> createSuccessResponse(Object data) { Map<String, Object> response = new HashMap<>(); response.put("code", 200); response.put("msg", "success"); response.put("data", data); return response; }
/** * 创建错误响应 */ private Map<String, Object> createErrorResponse(String message) { Map<String, Object> response = new HashMap<>(); response.put("code", 500); response.put("msg", message); response.put("data", null); return response; }}四、使用示例
4.1 示例1:导出用户数据
package com.example.demo.user;
import com.example.export.core.ExportDataProvider;import com.example.export.core.ExcelRowConverter;import com.example.export.core.GenericAsyncExportHandler;import com.example.export.service.ExportTaskService;import cn.hutool.core.date.DateUtil;import cn.hutool.json.JSONUtil;import lombok.Data;import org.apache.poi.ss.usermodel.*;import org.springframework.stereotype.Service;import org.springframework.web.bind.annotation.*;
import javax.annotation.Resource;import java.util.*;
/** * 用户数据导出示例 */@RestController@RequestMapping("/api/user")public class UserExportController {
@Resource private UserExportService userExportService;
@Resource private ExportTaskService exportTaskService;
@Resource private GenericAsyncExportHandler exportHandler;
/** * 创建导出任务 */ @PostMapping("/export") public Map<String, Object> exportUsers(@RequestBody UserQueryParams params) { // 检查并发限制 if (!exportTaskService.checkConcurrentLimit(params.getUserId(), 3)) { return createErrorResponse("导出任务过多,请稍后再试"); }
// 创建任务 String taskName = "用户数据导出_" + DateUtil.now(); String queryParams = JSONUtil.toJsonStr(params); String taskId = exportTaskService.createExportTask( taskName, "user_data", queryParams, params.getUserId() );
// 异步执行 exportHandler.executeExport( taskId, "user_data", userExportService.getDataProvider(), userExportService.getRowConverter(), queryParams );
return createSuccessResponse(taskId); }
private Map<String, Object> createSuccessResponse(Object data) { Map<String, Object> response = new HashMap<>(); response.put("code", 200); response.put("msg", "success"); response.put("data", data); return response; }
private Map<String, Object> createErrorResponse(String message) { Map<String, Object> response = new HashMap<>(); response.put("code", 500); response.put("msg", message); return response; }}
/** * 用户导出服务 */@Serviceclass UserExportService {
@Resource private UserMapper userMapper;
/** * 获取数据提供者 */ public ExportDataProvider<User> getDataProvider() { return new ExportDataProvider<User>() { @Override public int queryTotalCount(String params) { UserQueryParams queryParams = JSONUtil.toBean(params, UserQueryParams.class); return userMapper.countByParams(queryParams); }
@Override public List<User> queryData(String params, int offset, int limit) { UserQueryParams queryParams = JSONUtil.toBean(params, UserQueryParams.class); queryParams.setOffset(offset); queryParams.setLimit(limit); return userMapper.selectByParams(queryParams); } }; }
/** * 获取行转换器 */ public ExcelRowConverter<User> getRowConverter() { return new ExcelRowConverter<User>() { @Override public String[] getHeaders() { return new String[]{"用户ID", "用户名", "姓名", "邮箱", "手机号", "状态", "创建时间"}; }
@Override public int[] getColumnWidths() { return new int[]{3000, 4000, 3000, 5000, 4000, 2500, 5000}; }
@Override public void convertToRow(Workbook workbook, Row row, User user, int rowIndex) { CellStyle dataStyle = createDataStyle(workbook);
createCell(row, 0, user.getId(), dataStyle); createCell(row, 1, user.getUsername(), dataStyle); createCell(row, 2, user.getRealName(), dataStyle); createCell(row, 3, user.getEmail(), dataStyle); createCell(row, 4, user.getPhone(), dataStyle); createCell(row, 5, user.getStatus() == 1 ? "正常" : "禁用", dataStyle); createCell(row, 6, DateUtil.formatDateTime(user.getCreateTime()), dataStyle); }
private void createCell(Row row, int column, String value, CellStyle style) { Cell cell = row.createCell(column); cell.setCellValue(value != null ? value : ""); cell.setCellStyle(style); }
private CellStyle createDataStyle(Workbook workbook) { CellStyle style = workbook.createCellStyle(); style.setBorderBottom(BorderStyle.THIN); style.setBorderTop(BorderStyle.THIN); style.setBorderLeft(BorderStyle.THIN); style.setBorderRight(BorderStyle.THIN); style.setAlignment(HorizontalAlignment.LEFT); style.setVerticalAlignment(VerticalAlignment.CENTER); return style; } }; }}
/** * 用户实体 */@Dataclass User { private String id; private String username; private String realName; private String email; private String phone; private Integer status; private Date createTime;}
/** * 查询参数 */@Dataclass UserQueryParams { private String userId; // 当前用户ID private String username; private String realName; private Integer status; private Date startTime; private Date endTime; private Integer offset; private Integer limit;}4.2 示例2:导出带图片的商品数据
package com.example.demo.product;
import com.example.export.core.ExportDataProvider;import com.example.export.core.ExcelRowConverter;import com.example.export.core.GenericAsyncExportHandler;import com.example.export.service.ExportTaskService;import cn.hutool.http.HttpUtil;import lombok.Data;import lombok.extern.slf4j.Slf4j;import org.apache.poi.ss.usermodel.*;import org.apache.poi.ss.util.CellRangeAddress;import org.springframework.stereotype.Service;import org.springframework.web.bind.annotation.*;
import javax.annotation.Resource;import java.io.ByteArrayOutputStream;import java.io.InputStream;import java.net.URL;import java.util.concurrent.*;
/** * 商品导出服务(包含图片) */@Slf4j@Serviceclass ProductExportService {
@Resource private ProductMapper productMapper;
// 图片下载线程池 private final ExecutorService imageExecutor = new ThreadPoolExecutor( 10, 20, 60L, TimeUnit.SECONDS, new LinkedBlockingQueue<>(1000), new ThreadPoolExecutor.CallerRunsPolicy() );
/** * 获取数据提供者 */ public ExportDataProvider<Product> getDataProvider() { return new ExportDataProvider<Product>() { @Override public int queryTotalCount(String params) { return productMapper.countAll(); }
@Override public List<Product> queryData(String params, int offset, int limit) { return productMapper.selectPage(offset, limit); } }; }
/** * 获取行转换器(带图片) */ public ExcelRowConverter<Product> getRowConverter() { return new ExcelRowConverter<Product>() { @Override public String[] getHeaders() { return new String[]{"商品ID", "商品名称", "价格", "库存", "分类", "商品图片"}; }
@Override public int[] getColumnWidths() { return new int[]{3000, 6000, 3000, 3000, 4000, 5000}; }
@Override public void convertToRow(Workbook workbook, Row row, Product product, int rowIndex) { // 设置行高(图片需要更高的行) row.setHeight((short) 1200);
CellStyle dataStyle = createDataStyle(workbook);
createCell(row, 0, product.getId(), dataStyle); createCell(row, 1, product.getName(), dataStyle); createCell(row, 2, String.valueOf(product.getPrice()), dataStyle); createCell(row, 3, String.valueOf(product.getStock()), dataStyle); createCell(row, 4, product.getCategory(), dataStyle); createCell(row, 5, "", dataStyle);
// 异步下载并插入图片 if (product.getImageUrl() != null) { insertImageAsync(workbook, row.getSheet(), product.getImageUrl(), rowIndex); } }
/** * 异步插入图片 */ private void insertImageAsync(Workbook workbook, Sheet sheet, String imageUrl, int rowIndex) { Future<byte[]> future = imageExecutor.submit(() -> downloadImage(imageUrl));
try { // 5秒超时 byte[] imageBytes = future.get(5, TimeUnit.SECONDS);
if (imageBytes != null && imageBytes.length > 0) { // 添加图片到工作簿 int pictureIdx = workbook.addPicture(imageBytes, Workbook.PICTURE_TYPE_JPEG);
// 创建绘图对象 Drawing<?> drawing = sheet.createDrawingPatriarch();
// 设置图片位置(第6列,从当前行到下一行) ClientAnchor anchor = drawing.createAnchor( 0, 0, 0, 0, // dx1, dy1, dx2, dy2 5, rowIndex + 1, 6, rowIndex + 2 // col1, row1, col2, row2 );
anchor.setAnchorType(ClientAnchor.AnchorType.MOVE_AND_RESIZE);
// 创建图片 Picture picture = drawing.createPicture(anchor, pictureIdx); picture.resize(1.0); // 调整大小 } } catch (TimeoutException e) { log.warn("图片下载超时: {}", imageUrl); future.cancel(true); } catch (Exception e) { log.warn("插入图片失败: {}", imageUrl, e); } }
/** * 下载图片 */ private byte[] downloadImage(String imageUrl) { try { URL url = new URL(imageUrl); try (InputStream in = url.openStream(); ByteArrayOutputStream out = new ByteArrayOutputStream()) {
byte[] buffer = new byte[8192]; int bytesRead; while ((bytesRead = in.read(buffer)) != -1) { out.write(buffer, 0, bytesRead); } return out.toByteArray(); } } catch (Exception e) { log.warn("下载图片失败: {}", imageUrl, e); return null; } }
private void createCell(Row row, int column, String value, CellStyle style) { Cell cell = row.createCell(column); cell.setCellValue(value != null ? value : ""); cell.setCellStyle(style); }
private CellStyle createDataStyle(Workbook workbook) { CellStyle style = workbook.createCellStyle(); style.setBorderBottom(BorderStyle.THIN); style.setBorderTop(BorderStyle.THIN); style.setBorderLeft(BorderStyle.THIN); style.setBorderRight(BorderStyle.THIN); style.setAlignment(HorizontalAlignment.CENTER); style.setVerticalAlignment(VerticalAlignment.CENTER); return style; } }; }}
/** * 商品实体 */@Dataclass Product { private String id; private String name; private Double price; private Integer stock; private String category; private String imageUrl;}4.3 前端代码(Vue3 + Element Plus)
<template> <div class="export-container"> <!-- 导出按钮 --> <el-button type="primary" @click="handleExport"> <el-icon><Download /></el-icon> 导出数据 </el-button>
<!-- 导出历史按钮 --> <el-button @click="showExportHistory"> <el-icon><Document /></el-icon> 导出记录 </el-button>
<!-- 进度对话框 --> <el-dialog v-model="progressVisible" title="导出进度" width="500px" :close-on-click-modal="false" :close-on-press-escape="false" > <div class="progress-content"> <el-progress :percentage="taskInfo.progress" :status="getProgressStatus()" />
<div class="progress-info"> <p v-if="taskInfo.status === 0">等待处理...</p> <p v-else-if="taskInfo.status === 1"> 正在处理: {{ taskInfo.processedCount }} / {{ taskInfo.totalCount }} </p> <p v-else-if="taskInfo.status === 2" class="success"> 导出完成!文件大小: {{ formatFileSize(taskInfo.fileSize) }} </p> <p v-else-if="taskInfo.status === 3" class="error"> 导出失败: {{ taskInfo.errorMsg }} </p> </div> </div>
<template #footer> <el-button v-if="taskInfo.status === 2" type="primary" @click="downloadFile" > 下载文件 </el-button> <el-button @click="progressVisible = false">关闭</el-button> </template> </el-dialog>
<!-- 导出历史对话框 --> <el-dialog v-model="historyVisible" title="导出记录" width="800px" > <el-table :data="historyList" v-loading="historyLoading" style="width: 100%" > <el-table-column prop="taskName" label="任务名称" width="200" /> <el-table-column label="状态" width="100"> <template #default="{ row }"> <el-tag :type="getStatusType(row.status)"> {{ getStatusText(row.status) }} </el-tag> </template> </el-table-column> <el-table-column prop="progress" label="进度" width="100"> <template #default="{ row }"> {{ row.progress }}% </template> </el-table-column> <el-table-column prop="totalCount" label="记录数" width="100" /> <el-table-column label="文件大小" width="100"> <template #default="{ row }"> {{ formatFileSize(row.fileSize) }} </template> </el-table-column> <el-table-column prop="createTime" label="创建时间" width="180" /> <el-table-column label="操作" fixed="right"> <template #default="{ row }"> <el-button v-if="row.status === 2" link type="primary" @click="downloadHistoryFile(row.id)" > 下载 </el-button> <el-button v-if="row.status === 1" link type="primary" @click="viewProgress(row.id)" > 查看进度 </el-button> <el-button link type="danger" @click="deleteTask(row.id)" > 删除 </el-button> </template> </el-table-column> </el-table>
<el-pagination v-model:current-page="pagination.current" v-model:page-size="pagination.size" :total="pagination.total" layout="total, prev, pager, next" @current-change="loadHistoryList" /> </el-dialog> </div></template>
<script setup>import { ref, reactive } from 'vue'import { ElMessage } from 'element-plus'import { Download, Document } from '@element-plus/icons-vue'import axios from 'axios'
// 进度对话框const progressVisible = ref(false)const taskInfo = reactive({ taskId: '', status: 0, progress: 0, totalCount: 0, processedCount: 0, fileName: '', fileSize: 0, errorMsg: ''})
// 历史记录对话框const historyVisible = ref(false)const historyLoading = ref(false)const historyList = ref([])const pagination = reactive({ current: 1, size: 10, total: 0})
// 轮询定时器let pollingTimer = null
/** * 创建导出任务 */const handleExport = async () => { try { const params = { userId: 'current-user-id', // 从登录信息获取 // 其他查询参数... }
const response = await axios.post('/api/user/export', params)
if (response.data.code === 200) { taskInfo.taskId = response.data.data taskInfo.status = 0 taskInfo.progress = 0
progressVisible.value = true startPolling()
ElMessage.success('导出任务已创建') } else { ElMessage.error(response.data.msg || '创建任务失败') } } catch (error) { console.error('创建导出任务失败:', error) ElMessage.error('创建导出任务失败') }}
/** * 开始轮询 */const startPolling = () => { stopPolling() queryTaskStatus() pollingTimer = setInterval(queryTaskStatus, 2000)}
/** * 停止轮询 */const stopPolling = () => { if (pollingTimer) { clearInterval(pollingTimer) pollingTimer = null }}
/** * 查询任务状态 */const queryTaskStatus = async () => { try { const response = await axios.get(`/api/export/task/status/${taskInfo.taskId}`)
if (response.data.code === 200) { const data = response.data.data Object.assign(taskInfo, data)
// 任务完成或失败,停止轮询 if (data.status === 2 || data.status === 3) { stopPolling() } } } catch (error) { console.error('查询任务状态失败:', error) }}
/** * 下载文件 */const downloadFile = async () => { try { ElMessage.info('开始下载文件...')
const response = await axios.get( `/api/export/task/download/${taskInfo.taskId}`, { responseType: 'blob' } )
// 创建Blob const blob = new Blob([response.data], { type: 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet' })
// 创建下载链接 const url = window.URL.createObjectURL(blob) const link = document.createElement('a') link.href = url link.download = taskInfo.fileName link.click()
// 清理 window.URL.revokeObjectURL(url)
ElMessage.success('文件下载成功') progressVisible.value = false } catch (error) { console.error('下载文件失败:', error) ElMessage.error('下载文件失败') }}
/** * 显示导出历史 */const showExportHistory = () => { historyVisible.value = true pagination.current = 1 loadHistoryList()}
/** * 加载历史列表 */const loadHistoryList = async () => { try { historyLoading.value = true
const response = await axios.get('/api/export/task/list', { params: { current: pagination.current, size: pagination.size, createUser: 'current-user-id' // 从登录信息获取 } })
if (response.data.code === 200) { historyList.value = response.data.data.records pagination.total = response.data.data.total } } catch (error) { console.error('加载导出记录失败:', error) ElMessage.error('加载导出记录失败') } finally { historyLoading.value = false }}
/** * 下载历史文件 */const downloadHistoryFile = async (taskId) => { try { ElMessage.info('开始下载文件...')
const response = await axios.get( `/api/export/task/download/${taskId}`, { responseType: 'blob' } )
// 从响应头获取文件名 const contentDisposition = response.headers['content-disposition'] let fileName = '导出文件.xlsx' if (contentDisposition) { const match = contentDisposition.match(/filename\*=UTF-8''(.+)/) if (match) { fileName = decodeURIComponent(match[1]) } }
// 创建下载 const blob = new Blob([response.data]) const url = window.URL.createObjectURL(blob) const link = document.createElement('a') link.href = url link.download = fileName link.click() window.URL.revokeObjectURL(url)
ElMessage.success('文件下载成功') } catch (error) { console.error('下载文件失败:', error) ElMessage.error('下载文件失败') }}
/** * 查看进度 */const viewProgress = (taskId) => { taskInfo.taskId = taskId historyVisible.value = false progressVisible.value = true startPolling()}
/** * 删除任务 */const deleteTask = async (taskId) => { try { await ElMessageBox.confirm('确定要删除这个任务吗?', '提示', { type: 'warning' })
const response = await axios.delete(`/api/export/task/${taskId}`)
if (response.data.code === 200) { ElMessage.success('删除成功') loadHistoryList() } else { ElMessage.error(response.data.msg) } } catch (error) { if (error !== 'cancel') { console.error('删除任务失败:', error) ElMessage.error('删除任务失败') } }}
/** * 获取进度状态 */const getProgressStatus = () => { if (taskInfo.status === 2) return 'success' if (taskInfo.status === 3) return 'exception' return null}
/** * 获取状态文本 */const getStatusText = (status) => { const map = { 0: '等待中', 1: '处理中', 2: '已完成', 3: '失败', 4: '已过期' } return map[status] || '未知'}
/** * 获取状态类型 */const getStatusType = (status) => { const map = { 0: 'info', 1: 'warning', 2: 'success', 3: 'danger', 4: 'info' } return map[status] || 'info'}
/** * 格式化文件大小 */const formatFileSize = (bytes) => { if (!bytes || bytes === 0) return '-'
const k = 1024 const sizes = ['B', 'KB', 'MB', 'GB'] const i = Math.floor(Math.log(bytes) / Math.log(k))
return (bytes / Math.pow(k, i)).toFixed(2) + ' ' + sizes[i]}
// 组件卸载时清理定时器onUnmounted(() => { stopPolling()})</script>
<style scoped>.export-container { padding: 20px;}
.progress-content { padding: 20px 0;}
.progress-info { margin-top: 20px; text-align: center;}
.progress-info p { margin: 10px 0; font-size: 14px;}
.progress-info .success { color: #67c23a;}
.progress-info .error { color: #f56c6c;}</style>五、高级特性
5.1 定时清理过期文件
package com.example.export.task;
import cn.hutool.core.date.DateUtil;import cn.hutool.core.io.FileUtil;import com.baomidou.mybatisplus.core.conditions.query.LambdaQueryWrapper;import com.example.export.entity.ExportTask;import com.example.export.service.ExportTaskService;import lombok.extern.slf4j.Slf4j;import org.springframework.beans.factory.annotation.Value;import org.springframework.scheduling.annotation.Scheduled;import org.springframework.stereotype.Component;
import javax.annotation.Resource;import java.util.Date;import java.util.List;
/** * 导出文件清理任务 */@Slf4j@Componentpublic class ExportFileCleanTask {
@Resource private ExportTaskService exportTaskService;
@Value("${export.keep-days:7}") private int keepDays;
/** * 每天凌晨2点执行清理 */ @Scheduled(cron = "0 0 2 * * ?") public void cleanExpiredFiles() { log.info("开始清理过期导出文件,保留天数: {}", keepDays);
try { // 计算过期时间 Date expireDate = DateUtil.offsetDay(new Date(), -keepDays);
// 查询过期且已完成的任务 LambdaQueryWrapper<ExportTask> wrapper = new LambdaQueryWrapper<>(); wrapper.eq(ExportTask::getStatus, ExportTask.STATUS_COMPLETED) .lt(ExportTask::getFinishTime, expireDate);
List<ExportTask> expiredTasks = exportTaskService.list(wrapper);
int successCount = 0; int failCount = 0;
for (ExportTask task : expiredTasks) { try { // 删除文件 if (task.getFilePath() != null) { FileUtil.del(task.getFilePath()); }
// 更新任务状态为已过期 task.setStatus(ExportTask.STATUS_EXPIRED); task.setFilePath(null); exportTaskService.updateById(task);
successCount++; } catch (Exception e) { log.error("删除文件失败: {}", task.getFilePath(), e); failCount++; } }
log.info("清理完成,成功: {}, 失败: {}", successCount, failCount);
} catch (Exception e) { log.error("清理过期文件失败", e); } }}5.2 全局异常处理
package com.example.export.exception;
import lombok.extern.slf4j.Slf4j;import org.springframework.web.bind.annotation.ExceptionHandler;import org.springframework.web.bind.annotation.RestControllerAdvice;
import java.util.HashMap;import java.util.Map;
/** * 全局异常处理器 */@Slf4j@RestControllerAdvicepublic class GlobalExceptionHandler {
@ExceptionHandler(Exception.class) public Map<String, Object> handleException(Exception e) { log.error("系统异常", e);
Map<String, Object> response = new HashMap<>(); response.put("code", 500); response.put("msg", "系统异常: " + e.getMessage()); response.put("data", null);
return response; }
@ExceptionHandler(IllegalArgumentException.class) public Map<String, Object> handleIllegalArgumentException(IllegalArgumentException e) { log.warn("参数错误: {}", e.getMessage());
Map<String, Object> response = new HashMap<>(); response.put("code", 400); response.put("msg", e.getMessage()); response.put("data", null);
return response; }}5.3 并发控制拦截器
package com.example.export.interceptor;
import com.example.export.service.ExportTaskService;import lombok.extern.slf4j.Slf4j;import org.springframework.beans.factory.annotation.Value;import org.springframework.stereotype.Component;import org.springframework.web.servlet.HandlerInterceptor;
import javax.annotation.Resource;import javax.servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse;
/** * 导出并发控制拦截器 */@Slf4j@Componentpublic class ExportConcurrentInterceptor implements HandlerInterceptor {
@Resource private ExportTaskService exportTaskService;
@Value("${export.max-concurrent-per-user:3}") private int maxConcurrentPerUser;
@Override public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) { // 获取当前用户ID(从Session或Token中获取) String userId = getCurrentUserId(request);
if (userId == null) { return true; }
// 检查并发限制 if (!exportTaskService.checkConcurrentLimit(userId, maxConcurrentPerUser)) { log.warn("用户 {} 导出任务超过限制", userId); response.setStatus(429); // Too Many Requests return false; }
return true; }
private String getCurrentUserId(HttpServletRequest request) { // 这里需要根据实际情况从Session或JWT Token中获取用户ID return request.getHeader("User-Id"); }}六、性能优化
6.1 数据库索引优化
-- 任务状态查询索引CREATE INDEX idx_status_create_time ON export_task(status, create_time);
-- 用户任务查询索引CREATE INDEX idx_user_status ON export_task(create_user, status);
-- 过期任务清理索引CREATE INDEX idx_status_finish_time ON export_task(status, finish_time);6.2 Excel写入优化
// 使用SXSSFWorkbook流式写入SXSSFWorkbook workbook = new SXSSFWorkbook(100); // 内存中只保留100行workbook.setCompressTempFiles(true); // 压缩临时文件
// 禁用自动刷新sheet.setRandomAccessWindowSize(100);
// 完成后清理workbook.dispose();6.3 查询优化
// 使用游标方式查询大数据量@Select("SELECT * FROM user WHERE status = #{status}")@Options(resultSetType = ResultSetType.FORWARD_ONLY, fetchSize = 1000)@ResultType(User.class)void selectByStatus(@Param("status") Integer status, ResultHandler<User> handler);
// 在导出时使用userMapper.selectByStatus(1, context -> { User user = (User) context.getResultObject(); writeToExcel(user);});七、常见问题
Q1: 导出任务一直处于待处理状态?
原因:线程池队列已满或线程池未正确配置
解决方案:
- 检查线程池配置是否生效
- 查看日志是否有异常信息
- 确认
@EnableAsync注解已添加
Q2: 内存溢出怎么办?
解决方案:
- 使用
SXSSFWorkbook替代XSSFWorkbook - 减小批次大小
batchSize - 调整 JVM 参数
-Xmx
Q3: 图片下载超时导致导出变慢?
解决方案:
- 设置合理的超时时间(5-10秒)
- 增加图片下载线程池大小
- 图片失败不影响整体导出
Q4: 如何实现断点续传?
解决方案:
// 记录最后处理的offsettask.setLastOffset(offset);exportTaskService.updateById(task);
// 重试时从lastOffset开始int startOffset = task.getLastOffset() != null ? task.getLastOffset() : 0;Q5: 如何支持更多文件格式?
解决方案:
- CSV: 使用
OpenCSV库 - PDF: 使用
iText或Apache PDFBox - Word: 使用
Apache POI-OOXML
八、总结
本方案提供了一个完整、通用、可扩展的Excel异步导出解决方案,具有以下特点:
✅ 开箱即用:只需实现数据查询接口即可快速集成
✅ 高性能:分批处理、流式写入、异步执行
✅ 可扩展:泛型设计,支持任意数据类型
✅ 生产就绪:完善的异常处理、日志记录、监控
适用于各类需要导出大量数据的业务场景,如订单导出、用户数据导出、财务报表等。
评论