前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
工具
TVP
发布
社区首页 >专栏 >【MySQL】记一次MySQL崩溃修复案例,再也不用删库跑路了!!

【MySQL】记一次MySQL崩溃修复案例,再也不用删库跑路了!!

作者头像
冰河
发布2020-10-29 16:49:28
1.1K0
发布2020-10-29 16:49:28
举报
文章被收录于专栏:冰河技术

作者个人研发的在高并发场景下,提供的简单、稳定、可扩展的延迟消息队列框架,具有精准的定时任务和延迟队列处理功能。自开源半年多以来,已成功为十几家中小型企业提供了精准定时调度方案,经受住了生产环境的考验。为使更多童鞋受益,现给出开源框架地址:

https://github.com/sunshinelyz/mykit-delay

PS: 欢迎各位Star源码,也可以pr你牛逼哄哄的代码。

写在前面

越来越觉得互联网行业是个高危行业,动不动就删库跑路!这不,这几天一个朋友在一家公司当运维,在测试服务器上执行了一条错误的命令,导致MySQL数据库崩溃了,纠结了好几天也没解决问题。后来,我们一起“摆龙门阵”时聊起了这个话题,于是,我晚上下班后,在家又开始深入研究MySQL源码了!从根源上找出MySQL为啥会崩溃,于是有了这篇文章的总结。

问题描述

研究MySQL源代码,调试并压测MySQL源代码时,MySQL崩溃了!问题是它竟然崩溃了!而且还损坏了InnoDB文件!!还好是在调试环境下发生的,赶紧看看如何解决这个问题,经过一系列的查阅资料、验证、对比、MySQL源码调试跟踪、修复损坏的InnoDB文件、总结等流程,整理成此文,如果以后真的发生在线上的生产坏境,也不用担心是不是要跑路的问题了,可以分分钟搞定MySQL的崩溃问题了!!

查看错误日志,如下:

代码语言:javascript
复制
-----------------------------------------161108 23:36:45 mysqld_safe Starting mysqld daemon with 
databases from /usr/local/mysql/var2020-05-01 23:36:46 0 [Warning] TIMESTAMP with implicit DEFAULT 
value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation 
for more details).2020-05-01 23:36:46 5497 [Note] Plugin 'FEDERATED' is disabled.2020-05-01 23:36:46 

7f11c48e1720 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be 
removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's 
internal memory allocator.2020-05-01 23:36:46 5497 [Note] InnoDB: Using atomics to ref count buffer 
pool pages2020-05-01 23:36:46 5497 [Note] InnoDB: The InnoDB memory heap is disabled2020-05-01 
23:36:46 5497 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins2020-05-01 23:36:46 5497 
[Note] InnoDB: Memory barrier is not used2020-05-01 23:36:46 5497 [Note] InnoDB: Compressed tables 

use zlib 1.2.32020-05-01 23:36:46 5497 [Note] InnoDB: Using CPU crc32 instructions2020-05-01 23:36:46 5497 [Note] InnoDB: Initializing buffer pool, size = 16.0M2020-05-01 23:36:46 5497 [Note] InnoDB: 
Completed initialization of buffer poolInnoDB: Database page corruption on disk or a failedInnoDB: 
file read of page 5.InnoDB: You may have to recover from a backup.2020-05-01 23:36:46 7f11c48e1720 
InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 
7478d078000000050000000000000000000000000f271f4d000700000000000000000000000000000000001b4000000000000
000000200f20000000000000006000000000000002d000000000000002e000000000000002f0000000000000030000000000(
省略很多类似代码)InnoDB: End of page dump2020-05-01 23:36:46 7f11c48e1720 InnoDB: uncompressed page, 
stored checksum in field1 1954074744, calculated checksums for field1: crc32 993334256, innodb 
2046145943, none 3735928559, stored checksum in field2 1139795846, calculated checksums for field2: 

crc32 993334256, innodb 1606613742, none 3735928559, page LSN 0 254222157, low 4 bytes of LSN at page end 254221236, page number (if stored to page already) 5, space id (if created with >= MySQL-4.1.1 
and stored already) 0InnoDB: Page may be a transaction system pageInnoDB: Database page corruption on disk or a failedInnoDB: file read of page 5.InnoDB: You may have to recover from a backup.InnoDB: It 
is also possible that your operatingInnoDB: system has corrupted its own file cacheInnoDB: and 
rebooting your computer removes theInnoDB: error.InnoDB: If the corrupt page is an index pageInnoDB: 
you can also try to fix the corruptionInnoDB: by dumping, dropping, and reimportingInnoDB: the 
corrupt table. You can use CHECKInnoDB: TABLE to scan your table for corruption.InnoDB: See also 
http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.htmlInnoDB: about forcing 
recovery.InnoDB: Ending processing because of a corrupt database page.2020-05-01 23:36:46 
7f11c48e1720  InnoDB: Assertion failure in thread 139714288817952 in file buf0buf.cc line 4201InnoDB: We intentionally generate a memory trap.InnoDB: Submit a detailed bug report to 
http://bugs.mysql.com.InnoDB: If you get repeated assertion failures or crashes, evenInnoDB: 

immediately after the mysqld startup, there may beInnoDB: corruption in the InnoDB tablespace. Please refer toInnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.htmlInnoDB: about 
forcing recovery.03:36:46 UTC - mysqld got signal 6 ;This could be because you hit a bug. It is also possible that this binaryor one of the libraries it was linked against is corrupt, improperly 
built,or misconfigured. This error can also be caused by malfunctioning hardware.We will try our best 
to scrape up some info that will hopefully helpdiagnose the problem, but since we have already 
crashed,something is definitely wrong and this may 
fail.key_buffer_size=16777216read_buffer_size=262144max_used_connections=0max_threads=1000thread_coun
t=0connection_count=0It is possible that mysqld could use up tokey_buffer_size + (read_buffer_size + 
sort_buffer_size)*max_threads = 798063 K  bytes of memoryHope that's ok; if not, decrease some 
variables in the equation.Thread pointer: 0x0Attempting backtrace. You can use the following 
information to find outwhere mysqld died. If you see no messages after this, something wentterribly 
wrong...stack_bottom = 0 thread_stack 0x40000/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)
[0x8e64b5]/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x41b)
[0x652fbb]/lib64/libpthread.so.0(+0xf7e0)[0x7f11c44c77e0]/lib64/libc.so.6(gsignal+0x35)
[0x7f11c315d625]/lib64/libc.so.6(abort+0x175)

[0x7f11c315ee05]/usr/local/mysql/bin/mysqld[0xa585c5]/usr/local/mysql/bin/mysqld[0xa6c7b4]/usr/local/
mysql/bin/mysqld[0xa6cbc7]/usr/local/mysql/bin/mysqld[0xa5bce2]/usr/local/mysql/bin/mysqld[0xa1e2ba]/usr/local/mysql/bin/mysqld[0xa0bf60]/usr/local/mysql/bin/mysqld[0x95a427]/usr/local/mysql/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)
[0x58f788]/usr/local/mysql/bin/mysqld[0x6e4a36]/usr/local/mysql/bin/mysqld(_Z11plugin_initPiPPci+0xb3e)
[0x6e826e]/usr/local/mysql/bin/mysqld[0x582d85]/usr/local/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x4d8)
[0x587d18]/lib64/libc.so.6(__libc_start_main+0xfd)
[0x7f11c3149d5d]/usr/local/mysql/bin/mysqld[0x57a019]The manual page at 
http://dev.mysql.com/doc/mysql/en/crashing.html containsinformation that should help you find out 
what is causing the crash.161108 23:36:46 mysqld_safe mysqld from pid file 
/usr/local/mysql/var/VM_241_49_centos.pid 
ended------------------------------------------------------------------------------

问题分析

从日志中可以看出是innodb引擎出了问题。日志里提示到 http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html查看强制恢复的方法。在mysql的配置文件my.cnf里找到 [mysqld]字段下,添加 innodb_force_recovery=1:

代码语言:javascript
复制
[mysqld]innodb_force_recovery = 1

如果innodb_force_recovery = 1不生效,则可尝试2——6几个数字

然后重启mysql,重启成功。然后使用mysqldump或 pma 导出数据,执行修复操作等。修复完成后,把该参数注释掉,还原默认值0。

配置文件的参数:innodb_force_recovery

innodb_force_recovery影响整个InnoDB存储引擎的恢复状况。默认为0,表示当需要恢复时执行所有的恢复操作(即校验数据页/purge undo/insert buffer merge/rolling back&forward),当不能进行有效的恢复操作时,mysql有可能无法启动,并记录错误日志;

innodb_force_recovery可以设置为1-6,大的数字包含前面所有数字的影响。当设置参数值大于0后,可以对表进行select,create,drop操作,但insert,update或者delete这类操作是不允许的。

  • (SRV_FORCE_IGNORE_CORRUPT):忽略检查到的corrupt页。
  • (SRV_FORCE_NO_BACKGROUND):阻止主线程的运行,如主线程需要执行full purge操作,会导致crash。
  • (SRV_FORCE_NO_TRX_UNDO):不执行事务回滚操作。
  • (SRV_FORCE_NO_IBUF_MERGE):不执行插入缓冲的合并操作。
  • (SRV_FORCE_NO_UNDO_LOG_SCAN):不查看重做日志,InnoDB存储引擎会将未提交的事务视为已提交。
  • (SRV_FORCE_NO_LOG_REDO):不执行前滚的操作。

解决方案

一般修复方法参考:

第一种方法

建立一张新表:

代码语言:javascript
复制
create table demo_bak  #和原表结构一样,只是把INNODB改成了MYISAM。

把数据导进去

代码语言:javascript
复制
insert into demo_bak select * from demo;

删除掉原表:

代码语言:javascript
复制
drop table demo;

注释掉 innodb_force_recovery 之后,重启。

重命名:

代码语言:javascript
复制
rename table demo_bak to demo;

最后改回存储引擎:

代码语言:javascript
复制
alter table demo engine = innodb

第二种方法

另一个方法是使用mysqldump将表格导出,然后再导回到InnoDB表中。这两种方法的结果是相同的。 备份导出(包括结构和数据):

代码语言:javascript
复制
mysqldump -uroot -p123 test > test.sql

还原方法1:

代码语言:javascript
复制
use test;source test.sql

还原方法2(系统命令行):

代码语言:javascript
复制
mysql -uroot -p123 test < test.sql;

注意,CHECK TABLE命令在InnoDB数据库中基本上是没有用的。

第三种方法

1、配置my.cnf

配置innodb_force_recovery = 1或2——6几个数字,重启MySQL

2、导出数据脚本

代码语言:javascript
复制
mysqldump -uroot -p123 test > test.sql

导出SQL脚本。或者用Navicat将所有数据库/表导入到其他服务器的数据库中。 注意:这里的数据一定要备份成功。然后删除原数据库中的数据。

3、删除ib_logfile0、ib_logfile1、ibdata1

备份MySQL数据目录下的ib_logfile0、ib_logfile1、ibdata1三个文件,然后将这三个文件删除

4、配置my.cnf

将my.cnf中innodb_force_recovery = 1或2——6几个数字这行配置删除或者配置为innodb_force_recovery = 0,重启MySQL服务

5、将数据导入MySQL数据库

代码语言:javascript
复制
mysql -uroot -p123 test < test.sql; 或者用Navicat将备份的数据导入到数据库中。

此种方法下要注意的问题:

  • ib_logfile0、ib_logfile1、ibdata1这三个文件一定要先备份后删除;
  • 一定要确认原数据导出成功了
  • 当数据导出成功后,删除原数据库中的数据时,如果提示不能删除,可在命令行进入MySQL的数据目录,手动删除相关数据库的文件夹或者数据库文件夹下的数据表文件,前提是数据一定导出或备份成功。

本文参与 腾讯云自媒体同步曝光计划,分享自微信公众号。
原始发表:2020-05-15,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 冰河技术 微信公众号,前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体同步曝光计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
0 条评论
热度
最新
推荐阅读
目录
  • 写在前面
  • 问题描述
  • 问题分析
  • 解决方案
相关产品与服务
云数据库 SQL Server
腾讯云数据库 SQL Server (TencentDB for SQL Server)是业界最常用的商用数据库之一,对基于 Windows 架构的应用程序具有完美的支持。TencentDB for SQL Server 拥有微软正版授权,可持续为用户提供最新的功能,避免未授权使用软件的风险。具有即开即用、稳定可靠、安全运行、弹性扩缩等特点。
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档