-
Notifications
You must be signed in to change notification settings - Fork 1
/
Copy pathmysql.parallel.xml
192 lines (190 loc) · 12.1 KB
/
mysql.parallel.xml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V5.0//EN"
"/usr/share/xml/docbook/schema/dtd/4.5/docbookx.dtd" [
<!ENTITY article.author.xml SYSTEM "../common/article.author.xml">
<!ENTITY book.info.legalnotice.xml SYSTEM "../common/book.info.legalnotice.xml">
<!ENTITY book.info.abstract.xml SYSTEM "../common/book.info.abstract.xml">
]>
<article xml:base="http://netkiller.github.io/journal/" xmlns="http://docbook.org/ns/docbook" xml:lang="zh-cn">
<articleinfo>
<title>MySQL 大数据操作注意事项</title>
<subtitle>http://netkiller.github.io/journal/mysql.parallel.html</subtitle>
&article.author.xml;
&book.info.legalnotice.xml;
<abstract>
</abstract>
&book.info.abstract.xml;
<keywordset>
<keyword></keyword>
<keyword></keyword>
<keyword></keyword>
<keyword>
</keyword>
</keywordset>
<pubdate>2014-05-16</pubdate>
<release>$Id$</release>
</articleinfo>
<section>
<title>关于 delete</title>
<para>delete from mytable 必死无疑，你需要分批删除，尽量缩小每个批次删除的记录数，delete 是可以并行执行的，你可以同时运行多个删除操作</para>
<screen>
<![CDATA[
mysql> show processlist;
+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+
| 1 | event_scheduler | localhost | NULL | Daemon | 52 | Waiting for next activation | NULL |
| 115986 | dba | localhost | example | Query | 0 | NULL | show processlist |
| 117446 | dba | localhost | example | Query | 20 | updating | delete from mytable where OPEN_TIME like '2011.11.28%' |
| 117525 | dba | localhost | example | Query | 2 | updating | delete from mytable where OPEN_TIME like '2011.12.02%' |
| 117526 | dba | localhost | example | Query | 49 | updating | delete from mytable where OPEN_TIME like '2011.12.12%' |
| 117527 | dba | localhost | example | Query | 6 | updating | delete from mytable where OPEN_TIME like '2011.12.21%' |
| 117528 | dba | localhost | example | Query | 64 | updating | delete from mytable where OPEN_TIME like '2011.12.30%' |
| 117546 | dba | localhost | example | Query | 33 | updating | delete from mytable where OPEN_TIME like '2011.11.10%' |
+--------+-----------------+---------------------+-----------+---------+-------+-----------------------------+--------------------------------------------------------+
23 rows in set (0.00 sec)
]]>
</screen>
</section>
<section>
<title>关于 update</title>
<para>在电商领域常常遇到一个问题“调价”，经常需要调整一批商品的价格, 程序猿一条语句搞定有没有？</para>
<screen>
<![CDATA[
update goods set price=price+10 where category_id = xxx
]]>
</screen>
<para>在开发，测试环境是可以通过测试的，一旦部署到生产环境，必死无疑</para>
</section>
<section>
<title>关于创建索引</title>
<para>大表创建索引需要很久的时间，通常要经历 manage keys 与 copy to tmp table 的过程</para>
<screen>
<![CDATA[
mysql> show processlist;
+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+
| 1 | event_scheduler | localhost | NULL | Daemon | 47 | Waiting for next activation | NULL |
| 115986 | dba | localhost | example | Query | 0 | NULL | show processlist |
| 118814 | dba | 192.168.6.20:50459 | example | Query | 8 | copy to tmp table | ALTER TABLE `mytable` ADD INDEX `modifiy_time` (`MODIFY_TIME`) |
+--------+-----------------+---------------------+----------+---------+-------+-----------------------------+------------------------------------------------------------------+
17 rows in set (0.00 sec)
]]>
</screen>
<para>删除索引，也需要经理 copy to tmp table 过程，漫长的等待</para>
<screen>
<![CDATA[
mysql> show processlist;
+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+
| 1 | event_scheduler | localhost | NULL | Daemon | 11 | Waiting for next activation | NULL |
| 115986 | dba | localhost | example | Query | 0 | NULL | show processlist |
| 118814 | dba | 192.168.6.20:50459 | example | Query | 4 | copy to tmp table | ALTER TABLE `mytable` DROP INDEX `modifiy_time` |
+--------+-----------------+---------------------+--------------+---------+-------+-----------------------------+-------------------------------------------------+
17 rows in set (0.00 sec)
]]>
</screen>
<para>所以数据设计要深思熟虑，做到提前未雨绸缪，不要亡羊补牢</para>
</section>
<section>
<title>关于 OPTIMIZE</title>
<para>OPTIMIZE 的操作是将当前表复制到临时表操作后再删除当前表，最后将临时表改名</para>
<screen>
<![CDATA[
mysql> show processlist;
+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+
| 1 | event_scheduler | localhost | NULL | Daemon | 14 | Waiting for next activation | NULL |
| 115835 | dba | 192.168.6.20:49664 | example | Query | 9 | copy to tmp table | OPTIMIZE TABLE `mytable` |
| 115986 | dba | localhost | example | Query | 0 | NULL | show processlist |
+--------+-----------------+---------------------+---------------------------+---------+-------+-----------------------------+--------------------------+
17 rows in set (0.00 sec)
]]>
</screen>
</section>
<section>
<title>关于切换引擎</title>
<para>转换ENGINE从MyISAM到InnoDB会经历creating table然后copy to tmp table在修改表名几个阶段，过程非常缓慢</para>
<screen>
<![CDATA[
mysql> show processlist;
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
| 1 | event_scheduler | localhost | NULL | Daemon | 10 | Waiting for next activation | NULL |
| 3167 | dba | 192.168.6.20:56723 | example | Query | 2 | creating table | ALTER TABLE `mytable` ENGINE=InnoDB |
| 3172 | dba | localhost | example | Query | 0 | NULL | show processlist |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
18 rows in set (0.00 sec)
]]>
</screen>
<para>copy to tmp table 过程</para>
<screen>
<![CDATA[
mysql> show processlist;
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
| 1 | event_scheduler | localhost | NULL | Daemon | 21 | Waiting for next activation | NULL |
| 3167 | dba | 192.168.6.20:56723 | example | Query | 13 | copy to tmp table | ALTER TABLE `mytable` ENGINE=InnoDB |
| 3172 | dba | localhost | example | Query | 0 | NULL | show processlist |
+------+-----------------+---------------------+---------+---------+-------+-----------------------------+------------------------------------------+
18 rows in set (0.00 sec)
]]>
</screen>
<para>此时我们查看mysql data目录会看到临时表文件</para>
<screen>
<![CDATA[
# ll /var/lib/mysql/hx9999_real_history/
-rw-rw---- 1 mysql mysql 9522 May 16 17:17 #sql-c2f_c5f.frm
-rw-rw---- 1 mysql mysql 48 May 16 17:17 #sql-c2f_c5f.par
-rw-rw---- 1 mysql mysql 637534208 May 16 17:29 #sql-c2f_c5f#P#p0.ibd
-rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p1.ibd
-rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p2.ibd
-rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p3.ibd
-rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p4.ibd
-rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p5.ibd
-rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p6.ibd
-rw-rw---- 1 mysql mysql 180224 May 16 17:17 #sql-c2f_c5f#P#p7.ibd
]]>
</screen>
</section>
<section>
<title>确保SELECT不被受阻</title>
<para>使用各种手段保证select操作不被受阻，只要select一直可以查询网站前端就能提供80%的功能，一旦select受阻一切都是浮云。</para>
<para>保证 select 操作优先于其他操作</para>
<screen>
<![CDATA[
UPDATE [LOW_PRIORITY] [IGNORE] tbl_name
SET col_name1=expr1 [, col_name2=expr2 ...]
[WHERE where_definition]
[ORDER BY ...]
[LIMIT row_count]
]]>
</screen>
<para>update的时候增加 LOW_PRIORITY 参数，可以降低更新语句的优先级。</para>
<para>my.cnf</para>
<screen>
<![CDATA[
[mysqld]
low_priority_updates=1
]]>
</screen>
<para>或者启动是添加--low-priority-updates参数</para>
<para>全局开启</para>
<screen>
<![CDATA[
SET @@global.low_priority_updates = 1;
]]>
</screen>
<para>适用于本次会话连接</para>
<screen>
<![CDATA[
SET @@session.low_priority_updates = 1;
]]>
</screen>
</section>
</article>