惊心动魄-生产环境Mysql启动失败

021-06-05 14:27:07 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-06-05 14:27:07 25403 [Note] Plugin 'FEDERATED' is disabled.
06:27:07 UTC - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

key_buffer_size=4194304
read_buffer_size=4194304
max_used_connections=0
max_threads=3000
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 24621158 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x7ffff99f8040
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 7ffff99fb700 thread_stack 0x80000
/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x9024f5]
/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x3e8)[0x66f518]
/lib64/libpthread.so.0[0x3db500f7e0]
/usr/local/mysql/bin/mysqld(_Z18ha_resolve_by_nameP3THDPK19st_mysql_lex_stringb+0x70)[0x58db80]
/usr/local/mysql/bin/mysqld(_Z14open_table_defP3THDP11TABLE_SHAREj+0x2dcd)[0x787b3d]
/usr/local/mysql/bin/mysqld(_Z15get_table_shareP3THDP10TABLE_LISTPKcjjPij+0x189)[0x69dec9]
/usr/local/mysql/bin/mysqld[0x6a049c]
/usr/local/mysql/bin/mysqld(_Z10open_tableP3THDP10TABLE_LISTP18Open_table_context+0x846)[0x6a6966]
/usr/local/mysql/bin/mysqld(_Z11open_tablesP3THDPP10TABLE_LISTPjjP19Prelocking_strategy+0xbdc)[0x6a9ebc]
/usr/local/mysql/bin/mysqld(_Z20open_and_lock_tablesP3THDP10TABLE_LISTbjP19Prelocking_strategy+0x4e)[0x6aa10e]
/usr/local/mysql/bin/mysqld(_Z11plugin_initPiPPci+0xf32)[0x6fef52]
/usr/local/mysql/bin/mysqld[0x582fa0]
/usr/local/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x375)[0x583e55]
/lib64/libc.so.6(__libc_start_main+0x100)[0x3db4c1ed20]
/usr/local/mysql/bin/mysqld(__gxx_personality_v0+0x2d9)[0x578469]

Trying to get some variables.
Some pointers may be invalid and cause the dump to abort.
Query (0): is an invalid pointer
Connection ID (thread ID): 0
Status: NOT_KILLED

The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.
2021-06-05T06:27:07.478146Z mysqld_safe mysqld from pid file /data/mysql/mysqld.pid ended

mysql5.6安装路径：/usr/local/mysql
mysql5.7安装路径：/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64
根据上面异常信息，错误信息依旧是连接/usr/local/mysql。所以我修改/etc/profile的环境变量
export PATH=.$PATH:/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/bin

/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld_safe --defaults-file=/etc/my.cnf --user=mysql &

cd /usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/ 
/bin/sh ./bin/mysqld_safe --defaults-file=/etc/my.cnf --user=mysql &

[root@prod-slave mysql]# ps aux |grep mysql
root      6465  0.0  0.0 103328   852 pts/8    S+   15:15   0:00 grep mysql
root     30126  0.0  0.0 106252  1424 pts/11   S+   15:12   0:00 /bin/sh ./bin/mysqld_safe --defaults-file=/etc/my.cnf --user=mysql
mysql    30848 20.9  9.6 21781692 3160368 pts/11 Sl+ 15:12   0:38 /usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld --defaults-file=/etc/my.cnf --basedir=/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64 --datadir=/data/mysql/data/ --plugin-dir=/usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/lib/plugin --user=mysql --log-error=/data/logs/mysql/error.log --open-files-limit=10240 --pid-file=/data/mysql/mysqld.pid --socket=/data/mysql/mysql.sock --port=3306
[root@prod-slave mysql]# lsof -p 30126
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
sh      30126 root  cwd    DIR  252,1     4096 1310755 /usr/local/mysql-5.7.16-linux-glibc2.5-x86_64
sh      30126 root  rtd    DIR  252,1     4096       2 /
sh      30126 root  txt    REG  252,1   942200 1180875 /bin/bash
sh      30126 root  mem    REG  252,1   161776  262169 /lib64/ld-2.12.so
sh      30126 root  mem    REG  252,1  1930416  267634 /lib64/libc-2.12.so
sh      30126 root  mem    REG  252,1    23088  267913 /lib64/libdl-2.12.so
sh      30126 root  mem    REG  252,1   134792  267872 /lib64/libtinfo.so.5.7
sh      30126 root  mem    REG  252,1 99174448 1071533 /usr/lib/locale/locale-archive
sh      30126 root  mem    REG  252,1    26060 1053259 /usr/lib64/gconv/gconv-modules.cache
sh      30126 root    0u   CHR 136,11      0t0      14 /dev/pts/11
sh      30126 root    1u   CHR 136,11      0t0      14 /dev/pts/11
sh      30126 root    2u   CHR 136,11      0t0      14 /dev/pts/11
sh      30126 root  255r   REG  252,1    27186 1324136 /usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld_safe
[root@prod-slave mysql]# lsof -p 30848
COMMAND   PID  USER   FD   TYPE             DEVICE    SIZE/OFF       NODE NAME
mysqld  30848 mysql  cwd    DIR             252,17        4096    3932162 /data/mysql/data
mysqld  30848 mysql  rtd    DIR              252,1        4096          2 /
mysqld  30848 mysql  txt    REG              252,1   253303409    1324134 /usr/local/mysql-5.7.16-linux-glibc2.5-x86_64/bin/mysqld
mysqld  30848 mysql  mem    REG              252,1        5624     270897 /lib64/libaio.so.1.0.1
mysqld  30848 mysql  mem    REG              252,1      161776     262169 /lib64/ld-2.12.so
mysqld  30848 mysql  mem    REG              252,1     1930416     267634 /lib64/libc-2.12.so
mysqld  30848 mysql  mem    REG              252,1      146592     267906 /lib64/libpthread-2.12.so
mysqld  30848 mysql  mem    REG              252,1       23088     267913 /lib64/libdl-2.12.so
mysqld  30848 mysql  mem    REG              252,1       47760     267909 /lib64/librt-2.12.so
mysqld  30848 mysql  mem    REG              252,1      600048     262160 /lib64/libm-2.12.so
mysqld  30848 mysql  mem    REG              252,1       93320     267908 /lib64/libgcc_s-4.4.7-20120601.so.1
mysqld  30848 mysql  mem    REG              252,1       43928     267920 /lib64/libcrypt-2.12.so
mysqld  30848 mysql  mem    REG              252,1       12776     267919 /lib64/libfreebl3.so
mysqld  30848 mysql  mem    REG              252,1      217016     786460 /var/db/nscd/hosts
mysqld  30848 mysql  mem    REG              252,1       66432     267620 /lib64/libnss_files-2.12.so
mysqld  30848 mysql  mem    REG              252,1      989840    1071530 /usr/lib64/libstdc++.so.6.0.13
mysqld  30848 mysql    0r   CHR                1,3         0t0       3824 /dev/null
mysqld  30848 mysql    1w   REG             252,17       12824     262147 /data/logs/mysql/error.log
mysqld  30848 mysql    2w   REG             252,17       12824     262147 /data/logs/mysql/error.log
mysqld  30848 mysql    3u   REG             252,17          62     262163 /data/logs/mysql/binlog.index
mysqld  30848 mysql    4uW  REG             252,17  6777995264    3932167 /data/mysql/data/ibdata1
mysqld  30848 mysql    5u   REG              252,1           0     396354 /tmp/ibk9eQtl (deleted)
mysqld  30848 mysql    6u   REG