今天我们的zabbix-server机器上线半年时间，占用了500G空间，我一步步排查结果发现是/var/lib/mysql/下的libdata1文件过大，已经达到了300G。我立即想到了zabbix的数据库原因，zabbix的数据库他的表模式是共享表空间模式，随着数据增长，ibdata1 越来越大，性能方面会有影响，而且innodb把数据和索引都放在ibdata1下。
共享表空间模式：
InnoDB 默认会将所有的数据库InnoDB引擎的表数据存储在一个共享空间中：ibdata1，这样就感觉不爽，增删数据库的时候，ibdata1文件不会自动收缩，单个数据库的备份也将成为问题。通常只能将数据使用mysqldump 导出，然后再导入解决这个问题。
独立表空间模式：
优点：
1.每个表都有自已独立的表空间。
2.每个表的数据和索引都会存在自已的表空间中。
3.可以实现单表在不同的数据库中移动。
4.空间可以回收（drop/truncate table方式操作表空间不能自动回收）
5.对于使用独立表空间的表，不管怎麽删除，表空间的碎片不会太严重的影响性能，而且还有机会处理。
缺点：
单表增加比共享空间方式更大。
结论：
共享表空间在Insert操作上有一些优势，但在其它都没独立表空间表现好，所以我们要改成独立表空间。
当启用独立表空间时，请合理调整一下innodb_open_files 参数。
下面我们来讲下如何讲zabbix数据库修改成独立表空间模式
查看文件大小
l 查看系统空间利用率
[root@zabbix ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_zabbix-lv_root 1000G 468G 482G 50% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 485M 39M 421M 9% /boot
查看文件空间占用情况
[root@zabbix ~]# du / -h --max-depth=1
12K /.dbus
4.0K /selinux
1.7M /tmp
du: cannot access `/proc/7012/task/7012/fd/4': No suchfile or directory
du: cannot access `/proc/7012/task/7012/fdinfo/4': Nosuch file or directory
du: cannot access `/proc/7012/fd/4': No such file ordirectory
du: cannot access `/proc/7012/fdinfo/4': No such fileor directory
0 /proc
4.0K /media
4.0K /srv
8.0K /mnt
1.2G /root
0 /net
36K /home
39M /etc
462G /var
29M /boot
1.1G /backup
7.7M /bin
16K /lost+found
3.5G /usr
0 /misc
27M /lib64
0 /sys
8.0K /opt
15M /sbin
164K /dev
149M /lib
468G /
[root@zabbix ~]# cd /var/
[root@zabbix var]# du . -h --max-depth=1
504K ./spool
139G ./log
4.0K ./tmp
8.0K ./db
103M ./www
244K ./run
16K ./lock
4.0K ./crash
4.0K ./account
4.0K ./local
12K ./yp
33M ./cache
4.0K ./preserve
4.0K ./cvs
4.0K ./games
4.0K ./gdm
4.0K ./opt
8.0K ./empty
324G ./lib
4.0K ./nis
463G .
[root@zabbix var]# cd log/
[root@zabbix log]# ll
total 145373448
-rw-------. 1 root root 2368 Oct 24 2016 anaconda.ifcfg.log
-rw-------. 1 root root 35914 Oct 24 2016 anaconda.log
-rw-------. 1 root root 181001 Oct 24 2016 anaconda.program.log
-rw-------. 1 root root 300366 Oct 24 2016 anaconda.storage.log
-rw-------. 1 root root 114857 Oct 24 2016 anaconda.syslog
-rw-------. 1 root root 28547 Oct 24 2016 anaconda.xlog
-rw-------. 1 root root 115596 Oct 24 2016 anaconda.yum.log
drwxr-x---. 2 root root 4096 May 5 19:50 audit
-rw-r--r-- 1root root 2607 Jun 5 07:00 boot.log
-rw------- 1root utmp 0 Jun 1 03:31 btmp
-rw------- 1root utmp 768 May 5 18:38 btmp-20170601
drwxr-xr-x. 2 root root 4096 Oct 24 2016 ConsoleKit
-rw------- 1root root 24280 Jun 5 10:30 cron
-rw------- 1root root 128516 May 14 03:23 cron-20170514
-rw------- 1root root 129746 May 21 03:40 cron-20170521
-rw------- 1root root 128931 May 28 03:12 cron-20170528
-rw------- 1root root 129139 Jun 4 03:31 cron-20170604
drwxr-xr-x. 2 lp sys 4096 Aug 17 2013 cups
-rw-r--r-- 1root root 69891 Jun 5 07:00 dmesg
-rw-r--r-- 1root root 69891 May 29 07:00 dmesg.old
-rw------- 1root root 0 Jan 1 03:32 dracut.log
-rw-r--r--. 1 root root 345367 Oct 24 2016 dracut.log-20170101
drwxrwx--T. 2 root gdm 4096 Oct 24 2016 gdm
drwx------. 2 root root 4096 Jun 4 03:31 httpd
-rw-r--r--. 1 root root 145708 Jun 5 10:03 lastlog
-rw------- 1root root 1629 Jun 5 07:00 maillog
-rw------- 1root root 4465 May 13 08:19 maillog-20170514
-rw------- 1root root 5153 May 20 06:42 maillog-20170521
-rw------- 1root root 5153 May 27 06:30 maillog-20170528
-rw------- 1root root 5158 Jun 3 06:28 maillog-20170604
-rw------- 1root root 78017 Jun 5 10:05 messages
-rw------- 1root root 1317112 May 14 03:05 messages-20170514
-rw------- 1root root 1068946 May 21 03:06 messages-20170521
-rw------- 1root root 86245 May 28 03:05 messages-20170528
-rw------- 1root root 86065 Jun 4 03:06 messages-20170604
-rw-r-----. 1 mysql mysql 148857106048 Jun 5 10:30mysqld.log
drwxr-xr-x. 2 ntp ntp 4096 Nov 24 2013 ntpstats
-rw-r--r--. 1 root root 89 Oct 24 2016 pm-powersave.log
drwx------. 2 root root 4096 Aug 23 2010 ppp
drwxr-xr-x. 2 root root 4096 Oct 24 2016 prelink
drwxr-xr-x. 2 root root 4096 Jun 5 00:00 sa
drwx------. 3 root root 4096 Oct 24 2016 samba
-rw------- 1root root 855 Jun 5 10:03 secure
-rw------- 1root root 1855 May 12 18:54 secure-20170514
-rw------- 1root root 207 May 15 07:00 secure-20170521
-rw------- 1root root 1455 May 27 19:02 secure-20170528
-rw------- 1root root 1503 May 31 23:22 secure-20170604
-rw-------. 1 root root 0 Oct 24 2016 spice-vdagent.log
-rw------- 1root root 0 Jun 4 03:31 spooler
-rw------- 1root root 0 May 7 03:47 spooler-20170514
-rw------- 1root root 0 May 14 03:23 spooler-20170521
-rw------- 1root root 0 May 21 03:40 spooler-20170528
-rw------- 1root root 0 May 28 03:12 spooler-20170604
drwxr-x---. 2 root root 4096 Nov 23 2013 sssd
-rw-------. 1 root root 0 Oct 24 2016 tallylog
-rw-r--r--. 1 root root 0 Oct 24 2016 wpa_supplicant.log
-rw-rw-r--. 1 root utmp 325632 Jun 5 10:03 wtmp
-rw-r--r--. 1 root root 45892 Oct 25 2016 Xorg.0.log
-rw-r--r--. 1 root root 44521 Oct 24 2016 Xorg.0.log.old
-rw-r--r--. 1 root root 26974 Oct 24 2016 Xorg.9.log
-rw------- 1root root 1663 Apr 12 15:04 yum.log
-rw-------. 1 root root 6936 Dec 2 2016yum.log-20170101
drwxr-xr-x. 2 zabbix zabbix 4096 Oct 28 2016 zabbix
mysql.log日志文件占用空间过大，查看详细日志内容
[root@zabbix log]# tail mysqld.log
InnoDB: Your database may be corrupt or you may havecopied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB:http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: for more information.
170605 10:30:29 InnoDB: Error: page 14091764 log sequence number 462 2253153679
InnoDB: is in the future! Current system log sequencenumber 76 3553747713.
InnoDB: Your database may be corrupt or you may havecopied the InnoDB
InnoDB: tablespace but not the InnoDB log files. See
InnoDB:http://dev.mysql.com/doc/refman/5.1/en/forcing-innodb-recovery.html
InnoDB: for more information.
查看数据库文件夹硬盘空间使用情况
[root@zabbix log]# cd /var/lib/mysql
[root@zabbix mysql]# ll -h
total 324G
-rw-rw----. 1 mysql mysql 324G Jun 5 15:15 ibdata1
-rw-rw---- 1mysql mysql 5.0M Jun 5 15:15 ib_logfile0
-rw-rw---- 1mysql mysql 5.0M Jun 5 15:15 ib_logfile1
drwx------. 2 mysql mysql 4.0K Oct 28 2016 mysql
srwxrwxrwx 1mysql mysql 0 Jun 5 14:54 mysql.sock
drwx------. 2 mysql mysql 4.0K Jun 5 12:39 zabbix
-rw-r--r-- 1root root 251M Jun 5 15:15 zabbix02.sql
-rw-r--r-- 1root root 0 Jun 5 15:06 zabbix.sql
发现ibdata1 文件占用空间过大
2.清除zabbix数据库历史数据
1)查看哪些表的历史数据比较多
mysql> SELECT table_name AS "Tables",round(((data_length + index_length) / 1024 / 1024), 2) "Size in MB"FROM information_schema.TABLES WHERE table_schema = 'zabbix' ORDER BY(data_length + index_length) DESC;
+----------------------------+------------+
| Tables | Size in MB |
+----------------------------+------------+
| events | 181851.00 |
| history_uint | 92227.30 |
| history | 4590.27 |
| trends_uint | 3851.88 |
| history_str | 309.39 |
| trends | 156.75 |
| history_log | 89.17 |
| items | 36.41 |
| item_discovery | 6.23 |
| items_applications | 6.17 |
| alerts | 3.45 |
| triggers | 1.69 |
| p_w_picpaths | 1.53 |
| application_template | 0.63 |
| auditlog | 0.55 |
| acknowledges | 0.55 |
| functions | 0.48 |
| graphs | 0.47 |
| graphs_items | 0.44 |
| applications | 0.41 |
| hosts | 0.33 |
| hosts_groups | 0.27 |
| hostmacro | 0.22 |
| history_text | 0.22 |
| trigger_discovery | 0.20 |
| screens_items | 0.16 |
| profiles | 0.16 |
| graph_discovery | 0.13 |
| auditlog_details | 0.13 |
| hosts_templates | 0.11 |
| sysmaps_elements | 0.09 |
| host_discovery | 0.09 |
| interface | 0.09 |
| httptest | 0.08 |
| sysmaps | 0.08 |
| sysmaps_links | 0.06 |
| scripts | 0.06 |
| group_prototype | 0.06 |
| interface_discovery | 0.06 |
| drules | 0.05 |
| icon_mapping | 0.05 |
| users_groups | 0.05 |
| slideshows | 0.05 |
| media | 0.05 |
| icon_map | 0.05 |
| service_alarms | 0.05 |
| opgroup | 0.05 |
| slideshow_usrgrp | 0.05 |
| httptestitem | 0.05 |
| sysmaps_link_triggers | 0.05 |
| item_application_prototype | 0.05 |
| slideshow_user | 0.05 |
| application_prototype | 0.05 |
| maintenances_windows | 0.05 |
| slides | 0.05 |
| application_discovery | 0.05 |
| maintenances_hosts | 0.05 |
| httpstepitem | 0.05 |
| screens | 0.05 |
| config | 0.05 |
| trigger_depends | 0.05 |
| optemplate | 0.05 |
| maintenances_groups | 0.05 |
| sysmap_usrgrp | 0.05 |
| screen_usrgrp | 0.05 |
| opcommand_hst | 0.05 |
| opmessage_usr | 0.05 |
| actions | 0.05 |
| maintenances | 0.05 |
| sysmap_user | 0.05 |
| screen_user | 0.05 |
| opcommand_grp | 0.05 |
| services_links | 0.05 |
| dservices | 0.05 |
| opmessage_grp | 0.05 |
| rights | 0.05 |
| services | 0.03 |
| opmessage | 0.03 |
| usrgrp | 0.03 |
| sysmap_element_url | 0.03 |
| regexps | 0.03 |
| media_type | 0.03 |
| dhosts | 0.03 |
| proxy_history | 0.03 |
| item_condition | 0.03 |
| users | 0.03 |
| globalmacro | 0.03 |
| proxy_dhistory | 0.03 |
| mappings | 0.03 |
| groups | 0.03 |
| dchecks | 0.03 |
| proxy_autoreg_host | 0.03 |
| operations | 0.03 |
| expressions | 0.03 |
| group_discovery | 0.03 |
| opconditions | 0.03 |
| sessions | 0.03 |
| httpstep | 0.03 |
| conditions | 0.03 |
| services_times | 0.03 |
| escalations | 0.03 |
| autoreg_host | 0.03 |
| valuemaps | 0.03 |
| sysmap_url | 0.03 |
| graph_theme | 0.03 |
| opcommand | 0.03 |
| timeperiods | 0.02 |
| opinventory | 0.02 |
| globalvars | 0.02 |
| host_inventory | 0.02 |
| dbversion | 0.02 |
| housekeeper | 0.02 |
| ids | 0.02 |
+----------------------------+------------+
113 rows in set (7.31 sec)
可以看到events、history和history_uint这三个表的历史数据最多。
由于数据量太大，按照普通的方式delete数据的话基本上不太可能。
所以决定直接采用truncatetable的方式来快速清空这些表的数据，再使用mysqldump导出数据，删除共享表空间数据文件，重新导入数据。其中events空间过大，尝试truncatetable无果，果断抛弃
2）停止相关服务，避免写入数据
[root@zabbix ~]#/etc/init.d/zabbix_server stop
[root@zabbix ~]#/etc/init.d/httpd stop
3）清空历史数据
[root@zabbix ~]#mysql -uroot -p
mysql > use zabbix;
Database changed
mysql > truncate table history;
Query OK, 123981681 rows affected (0.23 sec)
mysql > optimize table history;
1 row in set (0.02 sec)
mysql > truncate table history_uint;
Query OK, 57990562 rows affected (0.12 sec)
mysql > optimize table history_uint;
1 row in set (0.03 sec)
3.备份数据库
[root@zabbix~]#mysqldump -uroot -p zabbix > /data/zabbix.sql
4.停止数据库并删除共享表空间数据文件
1)停止数据库
[root@zabbix ~]#/etc/init.d/mysqld stop
2）删除共享表空间数据文件
[root@zabbix ~]#cd /var/lib/mysql
[root@zabbix ~]#rm -rf ib*
[root@zabbix ~]#mkdir /tmp/zabbix
[root@zabbix ~]#mv zabbix/* /tmp/zabbix
5.增加innodb_file_per_table参数
[root@zabbix ~]#vi /etc/my.cnf
在[mysqld]下设置
innodb_file_per_table=1
6.启动mysql
[root@zabbix ~]#/etc/init.d/mysqld start
7.查看innodb_file_per_table参数是否生效
[root@zabbix ~]#mysql -uroot -p
mysql> show variables like '%per_table%';
+-----------------------+-------+
| Variable_name | Value |
+-----------------------+-------+
| innodb_file_per_table | ON |
+-----------------------+-------+
1 row in set (0.00 sec)
8.重新导入数据库
mysql >use zabbix;
mysql >source mysql.sql
9.最后，恢复相关服务进程
[root@zabbix ~]#/etc/init.d/zabbix_server start
[root@zabbix ~]#/etc/init.d/httpd start
恢复完服务之后，查看/分区的容量就下去了，之前是50%，处理完之后变成了2%。可见其成效
[root@zabbix ~]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_zabbix-lv_root 1000G 13G 936G 2% /
tmpfs 7.8G 0 7.8G 0% /dev/shm
/dev/sda1 485M 39M 421M 9% /boot
mysql> select table_name, (data_length+index_length)/1024/1024 as total_mb,table_rows from information_schema.tables where table_schema='zabbix' order bytable_rows;
+----------------------------+---------------+------------+
| table_name | total_mb | table_rows |
+----------------------------+---------------+------------+
| slideshows | 0.04687500 | 0 |
| globalvars | 0.01562500 | 0 |
| proxy_history | 0.03125000 | 0 |
| icon_map | 0.04687500 | 0 |
| sysmaps_links | 0.06250000 | 0 |
| service_alarms | 0.04687500 | 0 |
| opinventory | 0.01562500 | 0 |
| slideshow_usrgrp | 0.04687500 | 0 |
| proxy_dhistory | 0.03125000 | 0 |
| sysmaps_link_triggers | 0.04687500 | 0 |
| proxy_autoreg_host | 0.03125000 | 0 |
| application_prototype | 0.04687500 | 0 |
| maintenances_windows | 0.04687500 | 0 |
| item_application_prototype | 0.04687500 | 0 |
| host_inventory | 0.01562500 | 0 |
| slideshow_user | 0.04687500 | 0 |
| application_discovery | 0.04687500 | 0 |
| maintenances_hosts | 0.04687500 | 0 |
| opconditions | 0.03125000 | 0 |
| slides | 0.04687500 | 0 |
| maintenances_groups | 0.04687500 | 0 |
| sysmap_usrgrp | 0.04687500 | 0 |
| screen_usrgrp | 0.04687500 | 0 |
| opcommand_hst | 0.04687500 | 0 |
| maintenances | 0.04687500 | 0 |
| housekeeper | 0.01562500 | 0 |
| sysmap_user | 0.04687500 | 0 |
| screen_user | 0.04687500 | 0 |
| opcommand_grp | 0.04687500 | 0 |
| services_times | 0.03125000 | 0 |
| escalations | 0.03125000 | 0 |
| sysmap_url | 0.03125000 | 0 |
| opcommand | 0.03125000 | 0 |
| services_links | 0.04687500 | 0 |
| sysmap_element_url | 0.03125000 | 0 |
| icon_mapping | 0.04687500 | 0 |
| timeperiods | 0.01562500 | 0 |
| services | 0.03125000 | 0 |
| httptest | 0.07812500 | 1 |
| sysmaps_elements | 0.09375000 | 1 |
| dbversion | 0.01562500 | 1 |
| sysmaps | 0.07812500 | 1 |
| config | 0.04687500 | 1 |
| autoreg_host | 0.03125000 | 1 |
| globalmacro | 0.03125000 | 2 |
| optemplate | 0.04687500 | 2 |
| opmessage_usr | 0.04687500 | 2 |
| graph_theme | 0.03125000 | 2 |
| media | 0.04687500 | 3 |
| users_groups | 0.04687500 | 3 |
| httptestitem | 0.04687500 | 3 |
| scripts | 0.06250000 | 3 |
| dchecks | 0.03125000 | 3 |
| opgroup | 0.04687500 | 3 |
| users | 0.03125000 | 3 |
| httpstep | 0.03125000 | 3 |
| regexps | 0.03125000 | 3 |
| media_type | 0.03125000 | 3 |
| drules | 0.04687500 | 3 |
| expressions | 0.03125000 | 4 |
| opmessage_grp | 0.04687500 | 4 |
| usrgrp | 0.03125000 | 5 |
| opmessage | 0.03125000 | 6 |
| httpstepitem | 0.04687500 | 9 |
| actions | 0.04687500 | 9 |
| dhosts | 0.03125000 | 12 |
| operations | 0.03125000 | 14 |
| dservices | 0.04687500 | 14 |
| group_prototype | 0.06250000 | 15 |
| conditions | 0.03125000 | 17 |
| rights | 0.04687500 | 17 |
| group_discovery | 0.03125000 | 19 |
| valuemaps | 0.03125000 | 21 |
| screens | 0.04687500 | 24 |
| groups | 0.03125000 | 31 |
| item_condition | 0.03125000 | 31 |
| sessions | 0.03125000 | 39 |
| ids | 0.01562500 | 46 |
| trigger_depends | 0.04687500 | 51 |
| mappings | 0.03125000 | 100 |
| hosts_templates | 0.12500000 | 181 |
| p_w_picpaths | 1.53125000 | 192 |
| host_discovery | 0.09375000 | 361 |
| interface_discovery | 0.06250000 | 515 |
| hosts | 0.32812500 | 516 |
| profiles | 0.17187500 | 659 |
| interface | 0.09375000 | 691 |
| auditlog_details | 0.12500000 | 707 |
| events | 0.17187500 | 759 |
| screens_items | 0.15625000 | 805 |
| history_text | 0.18750000 | 1201 |
| graph_discovery | 0.12500000 | 1217 |
| graphs | 0.46875000 | 1330 |
| hosts_groups | 0.26562500 | 1381 |
| hostmacro | 0.21875000 | 1569 |
| trigger_discovery | 0.20312500 | 2017 |
| acknowledges | 0.54687500 | 2408 |
| auditlog | 0.53125000 | 2546 |
| graphs_items | 0.43750000 | 2706 |
| triggers | 0.68750000 | 2900 |
| functions | 0.50000000 | 3134 |
| application_template | 0.51562500 | 4364 |
| applications | 0.42187500 | 4512 |
| history | 0.46875000 | 4780 |
| alerts | 3.46875000 | 6242 |
| item_discovery | 5.54687500 | 26567 |
| items_applications | 6.54687500 | 30181 |
| items | 24.12500000 | 34425 |
| history_uint | 12.03125000 | 123327 |
| history_log | 88.67187500 | 479448 |
| history_str | 76.17187500 | 743970 |
| trends | 122.68750000 | 2023329 |
| trends_uint | 3076.00000000 | 51097289 |
+----------------------------+---------------+------------+
113 rows in set (0.55 sec)
备注：truncate table events，太久没看到效果，尝试用delete shell脚本
#!/bin/bash
#delete events table
# edit by john zheng 20170606
i=1
while [ $i -lt events_id最大值 ];
do
echo "$i"
mysql -uroot -p密码 -e"
use zabbix;
delete from events where eventid < $i;"
i=`expr $i '+' 100000`
done
尝试失败，时间更久，直接放弃，只保留events创建表语句
mysql> show create table events;
+--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table |
+--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| events | CREATE TABLE `events` (
`eventid` bigint(20) unsigned NOT NULL,
`source` int(11) NOT NULL DEFAULT '0',
`object` int(11) NOT NULL DEFAULT '0',
`objectid` bigint(20) unsigned NOT NULL DEFAULT '0',
`clock` int(11) NOT NULL DEFAULT '0',
`value` int(11) NOT NULL DEFAULT '0',
`acknowledged` int(11) NOT NULL DEFAULT '0',
`ns` int(11) NOT NULL DEFAULT '0',
PRIMARY KEY (`eventid`),
KEY `events_1` (`source`,`object`,`objectid`,`clock`),
KEY `events_2` (`source`,`object`,`clock`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin |
+--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
