chchang/hugo_backup
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

update some content

Browse Source
This commit is contained in:
Eric Chang
2021-10-28 14:15:01 +08:00
parent b8600ab259
commit 27b5a11382
163 changed files with 951 additions and 922 deletions
Download Patch File Download Diff File Expand all files Collapse all files

58
public/post/postgresql-backup-restore-using-zfs-snapshot/index.html
View File

@@ -45,9 +45,9 @@
<link rel="stylesheet" href="https://use.fontawesome.com/releases/v5.3.1/css/all.css" integrity="sha384-mzrmE5qonljUremFsqc01SB46JvROS7bZs3IO2EmfFsd15uHvIt+Y8vEf7N7fWAU" crossorigin="anonymous">
<link href="https://h.cowbay.org/css/style.css?v=1634607506" rel="stylesheet" id="theme-stylesheet" type='text/css' media='all'>
<link href="https://h.cowbay.org/css/style.css?v=1635401692" rel="stylesheet" id="theme-stylesheet" type='text/css' media='all'>
<link href="https://h.cowbay.org/css/custom.css?v=1634607506" rel="stylesheet" type='text/css' media='all'>
<link href="https://h.cowbay.org/css/custom.css?v=1635401692" rel="stylesheet" type='text/css' media='all'>
<link rel="shortcut icon" href="https://h.cowbay.org/img/favicon.ico" type="image/x-icon">
<link rel="icon" href="https://h.cowbay.org/img/favicon.ico" type="image/x-icon">
@@ -310,7 +310,7 @@ if (!doNotTrack) {
<h3 id="建立測試資料庫tablesnapshot">建立測試資料庫、TABLE、snapshot</h3>
<h4 id="資料庫現況">資料庫現況</h4>
<p>只有系統預設的DB，沒有其他多的東西</p>
<pre tabindex="0"><code>postgres@hqdc034:~$ psql -c '\l'
<pre><code>postgres@hqdc034:~$ psql -c '\l'
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
@@ -324,7 +324,7 @@ if (!doNotTrack) {
postgres@hqdc034:~$ du -sh /zp/database/10/main/
232M /zp/database/10/main/
</code></pre><h4 id="建立第一次的快照">建立第一次的快照</h4>
<pre tabindex="0"><code>2019-09-06 09:03:46 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
<pre><code>2019-09-06 09:03:46 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
no datasets available
2019-09-06 09:03:53 [changch@hqdc034 ~]$ sudo zfs snapshot zp/database@init_db_no_demo
2019-09-06 09:04:09 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
@@ -332,13 +332,13 @@ NAME USED AVAIL REFER MOUNTPOINT
zp/database@init_db_no_demo 0 - 231M -
2019-09-06 09:04:15 [changch@hqdc034 ~]$
</code></pre><h4 id="建立倒回測試資料庫-demo">建立、倒回測試資料庫 demo</h4>
<pre tabindex="0"><code>postgres@hqdc034:~$ createdb demo
<pre><code>postgres@hqdc034:~$ createdb demo
postgres@hqdc034:~$ psql demo &lt; /home/changch/Downloads/demo.sql
SET
SET
略...
</code></pre><p>再檢查一次資料庫的狀況，看到 demo DB出現了，資料庫目錄也變大了</p>
<pre tabindex="0"><code>postgres@hqdc034:~$ psql -c '\l'
<pre><code>postgres@hqdc034:~$ psql -c '\l'
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+-------------+-------------+-----------------------
@@ -355,25 +355,25 @@ postgres@hqdc034:~$ du -sh /zp/database/10/main/
postgres@hqdc034:~$
</code></pre><h4 id="建立第二次快照">建立第二次快照</h4>
<p>這次的快照，將包含剛剛倒回的 demo DB，但是不包含等下才要建立的測試 table</p>
<pre tabindex="0"><code>2019-09-06 09:16:01 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
<pre><code>2019-09-06 09:16:01 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zp/database@init_db_no_demo 250K - 231M -
zp/database@demo_db_just_restore 0 - 2.08G -
2019-09-06 09:16:04 [changch@hqdc034 ~]$
</code></pre><h4 id="建立測試-table">建立測試 table</h4>
<pre tabindex="0"><code>postgres@hqdc034:~$ psql -c 'create table test ( a int, b varchar(50) );'
<pre><code>postgres@hqdc034:~$ psql -c 'create table test ( a int, b varchar(50) );'
CREATE TABLE
postgres@hqdc034:~$
</code></pre><h4 id="建立第三次快照">建立第三次快照</h4>
<p>這次快照，只有建立 test table ，但是裡面沒有資料</p>
<pre tabindex="0"><code>2019-09-06 09:18:34 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
<pre><code>2019-09-06 09:18:34 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zp/database@init_db_no_demo 250K - 231M -
zp/database@demo_db_just_restore 138K - 2.08G -
zp/database@demo_db_create_test_table_but_no_data 0 - 2.08G -
2019-09-06 09:18:36 [changch@hqdc034 ~]$
</code></pre><h4 id="在test-table-插入100萬筆資料">在test table 插入100萬筆資料</h4>
<pre tabindex="0"><code>postgres@hqdc034:~$ psql -c 'with aa as ( select * from generate_series (1,1000000) a ) insert into test select aa.a, md5(aa.a::varchar) from aa;'
<pre><code>postgres@hqdc034:~$ psql -c 'with aa as ( select * from generate_series (1,1000000) a ) insert into test select aa.a, md5(aa.a::varchar) from aa;'
INSERT 0 1000000
postgres@hqdc034:~$ psql -c 'select count(*) from test;'
count
@@ -384,7 +384,7 @@ postgres@hqdc034:~$ psql -c 'select count(*) from test;'
postgres@hqdc034:~$
</code></pre><h4 id="建立第四次快照">建立第四次快照</h4>
<p>test table 內有 1000000 筆資料</p>
<pre tabindex="0"><code>2019-09-06 09:18:36 [changch@hqdc034 ~]$ sudo zfs snapshot zp/database@demo_db_test_table_with_1M_rows
<pre><code>2019-09-06 09:18:36 [changch@hqdc034 ~]$ sudo zfs snapshot zp/database@demo_db_test_table_with_1M_rows
2019-09-06 09:21:08 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zp/database@init_db_no_demo 250K - 231M -
@@ -393,7 +393,7 @@ zp/database@demo_db_create_test_table_but_no_data 116K - 2.08G -
zp/database@demo_db_test_table_with_1M_rows 0 - 2.15G -
2019-09-06 09:21:09 [changch@hqdc034 ~]$
</code></pre><h4 id="再次插入-100萬筆資料">再次插入 100萬筆資料</h4>
<pre tabindex="0"><code>postgres@hqdc034:~$ time psql -c 'with aa as ( select * from generate_series (1,1000000) a ) insert into test select aa.a, md5(aa.a::varchar) from aa;'
<pre><code>postgres@hqdc034:~$ time psql -c 'with aa as ( select * from generate_series (1,1000000) a ) insert into test select aa.a, md5(aa.a::varchar) from aa;'
INSERT 0 1000000
real 0m4.276s
@@ -408,7 +408,7 @@ postgres@hqdc034:~$ psql -c 'select count(*) from test;'
postgres@hqdc034:~$
</code></pre><h4 id="建立第五次快照">建立第五次快照</h4>
<p>現在 test table 有 200萬筆資料了</p>
<pre tabindex="0"><code>2019-09-06 09:21:09 [changch@hqdc034 ~]$ sudo zfs snapshot zp/database@demo_db_test_table_with_2M_rows
<pre><code>2019-09-06 09:21:09 [changch@hqdc034 ~]$ sudo zfs snapshot zp/database@demo_db_test_table_with_2M_rows
2019-09-06 09:22:29 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zp/database@init_db_no_demo 250K - 231M -
@@ -418,7 +418,7 @@ zp/database@demo_db_test_table_with_1M_rows 218K - 2.15G -
zp/database@demo_db_test_table_with_2M_rows 0 - 2.23G -
2019-09-06 09:22:30 [changch@hqdc034 ~]$
</code></pre><h4 id="玩大點直接湊滿1000萬筆資料好了">玩大點，直接湊滿1000萬筆資料好了</h4>
<pre tabindex="0"><code>postgres@hqdc034:~$ time psql -c 'with aa as ( select * from generate_series (1,8000000) a ) insert into test select aa.a, md5(aa.a::varchar) from aa;'
<pre><code>postgres@hqdc034:~$ time psql -c 'with aa as ( select * from generate_series (1,8000000) a ) insert into test select aa.a, md5(aa.a::varchar) from aa;'
INSERT 0 8000000
real 0m32.172s
@@ -432,7 +432,7 @@ postgres@hqdc034:~$ psql -c 'select count(*) from test;'
postgres@hqdc034:~$
</code></pre><h4 id="建立第六次快照">建立第六次快照</h4>
<p>10M rows in test table</p>
<pre tabindex="0"><code>2019-09-06 09:22:30 [changch@hqdc034 ~]$ sudo zfs snapshot zp/database@demo_db_test_table_with_10M_rows
<pre><code>2019-09-06 09:22:30 [changch@hqdc034 ~]$ sudo zfs snapshot zp/database@demo_db_test_table_with_10M_rows
2019-09-06 09:25:18 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zp/database@init_db_no_demo 250K - 231M -
@@ -443,13 +443,13 @@ zp/database@demo_db_test_table_with_2M_rows 530K - 2.23G -
zp/database@demo_db_test_table_with_10M_rows 163K - 2.97G -
2019-09-06 09:25:21 [changch@hqdc034 ~]$
</code></pre><p>到1000萬筆資料為止，現在資料庫大小是這樣</p>
<pre tabindex="0"><code>postgres@hqdc034:~$ du -sh /zp/database/10/main/
<pre><code>postgres@hqdc034:~$ du -sh /zp/database/10/main/
3.0G /zp/database/10/main/
postgres@hqdc034:~$
</code></pre><hr>
<h3 id="還原測試">還原測試</h3>
<p>最後一次做快照的時候，demo DB 裡面有一千萬筆資料，現在來砍掉500萬筆</p>
<pre tabindex="0"><code>postgres@hqdc034:~$ time psql -c 'delete from test where a &gt; 5000000;'
<pre><code>postgres@hqdc034:~$ time psql -c 'delete from test where a &gt; 5000000;'
DELETE 3000000
real 0m7.844s
@@ -472,14 +472,14 @@ postgres@hqdc034:~$
<p>假設剛剛這個刪除是錯誤的動作，我要回到1000萬資料的狀態，就可以用zfs rollback 來達成</p>
<h4 id="第一次還原">第一次還原</h4>
<p>目標是還原到包含1000萬筆資料的狀態(現在是700萬筆)</p>
<pre tabindex="0"><code>2019-09-06 09:25:21 [changch@hqdc034 ~]$ sudo service postgresql stop
<pre><code>2019-09-06 09:25:21 [changch@hqdc034 ~]$ sudo service postgresql stop
* Stopping PostgreSQL 10 database server [ OK ]
2019-09-06 10:14:12 [changch@hqdc034 ~]$ sudo zfs rollback -r zp/database@demo_db_test_table_with_10M_rows
2019-09-06 10:14:28 [changch@hqdc034 ~]$ sudo service postgresql start
* Starting PostgreSQL 10 database server [ OK ]
2019-09-06 10:14:57 [changch@hqdc034 ~]$
</code></pre><p>檢查一下</p>
<pre tabindex="0"><code>postgres@hqdc034:~$ time psql -c 'select count(*) from test;'
<pre><code>postgres@hqdc034:~$ time psql -c 'select count(*) from test;'
count
----------
10000000
@@ -493,7 +493,7 @@ postgres@hqdc034:~$
</code></pre><p>沒錯，又回到1000萬筆資料的狀態了</p>
<p>要注意的是，如果回到更之前的狀態，在該狀態之後的快照將會被清除，除非你先做clone
比如我現在要回到 200萬筆的狀態，那1000萬筆資料的快照就會被刪除</p>
<pre tabindex="0"><code>2019-09-06 10:17:32 [changch@hqdc034 ~]$ sudo service postgresql stop
<pre><code>2019-09-06 10:17:32 [changch@hqdc034 ~]$ sudo service postgresql stop
* Stopping PostgreSQL 10 database server [ OK ]
2019-09-06 10:18:50 [changch@hqdc034 ~]$ sudo zfs rollback -r zp/database@demo_db_test_table_with_2M_rows
2019-09-06 10:18:57 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
@@ -519,7 +519,7 @@ sys 0m0.008s
postgres@hqdc034:~$
</code></pre><p>啊，我剛剛應該先clone的&hellip;.
沒關係，我們再做一次，新增800萬筆資料，湊齊1000萬筆，然後快照</p>
<pre tabindex="0"><code>postgres@hqdc034:~$ time psql -c 'with aa as ( select * from generate_series (1,8000000) a ) insert into test select aa.a, md5(aa.a::varchar) from aa;'
<pre><code>postgres@hqdc034:~$ time psql -c 'with aa as ( select * from generate_series (1,8000000) a ) insert into test select aa.a, md5(aa.a::varchar) from aa;'
INSERT 0 8000000
real 0m35.662s
@@ -536,7 +536,7 @@ user 0m0.024s
sys 0m0.008s
postgres@hqdc034:~$
</code></pre><p>做快照</p>
<pre tabindex="0"><code>2019-09-06 10:19:17 [changch@hqdc034 ~]$ sudo zfs snapshot zp/database@demo_db_test_table_with_10M_rows
<pre><code>2019-09-06 10:19:17 [changch@hqdc034 ~]$ sudo zfs snapshot zp/database@demo_db_test_table_with_10M_rows
2019-09-06 10:22:59 [changch@hqdc034 ~]$ sudo zfs list -t snapshot
NAME USED AVAIL REFER MOUNTPOINT
zp/database@init_db_no_demo 250K - 231M -
@@ -547,11 +547,11 @@ zp/database@demo_db_test_table_with_2M_rows 56.4M - 2.23G -
zp/database@demo_db_test_table_with_10M_rows 0 - 1.81G -
2019-09-06 10:23:02 [changch@hqdc034 ~]$
</code></pre><p>接著來測試看看 clone snapshot，這是基本的說明</p>
<pre tabindex="0"><code>Clones can only be created from a snapshot and a snapshot can not
<pre><code>Clones can only be created from a snapshot and a snapshot can not
be deleted until you delete the clone that is based on this snapshot.
To create a clone, use the zfs clone command.
</code></pre><p>clone 會做出一份跟clone來源一模一樣的資料，在快照模式下，資料是唯讀的，clone出來後，就可以做異動。但是不能刪除clone來源的快照，會提示錯誤。</p>
<pre tabindex="0"><code>2019-09-06 10:28:31 [changch@hqdc034 ~]$ sudo zfs clone zp/database@demo_db_test_table_with_10M_rows zp/database/clone_with_10M_rows
<pre><code>2019-09-06 10:28:31 [changch@hqdc034 ~]$ sudo zfs clone zp/database@demo_db_test_table_with_10M_rows zp/database/clone_with_10M_rows
2019-09-06 10:29:21 [changch@hqdc034 ~]$ sudo zfs list
NAME USED AVAIL REFER MOUNTPOINT
zp 3.08G 231G 22K /zp
@@ -561,18 +561,18 @@ zp/database/clone_with_10M_rows 0 231G 1.81G /zp/database/clone_with_10
</code></pre><p>可以看到做了clone之後，多了一個 zfs dataset
試試看把資料庫路徑直接改到這個新做的dataset 看看能不能啟動資料庫
修改 /etc/postgresql/10/main/postgresql.conf，然後重起postgresql</p>
<pre tabindex="0"><code>#data_directory = '/var/lib/postgresql/10/main' # use data in another directory
<pre><code>#data_directory = '/var/lib/postgresql/10/main' # use data in another directory
#data_directory = '/zp/database/10/main'
data_directory = '/zp/database/clone_with_10M_rows/10/main'
</code></pre><p><strong>啟動有比較久一點</strong> 而且好像沒成功啟動</p>
<pre tabindex="0"><code>2019-09-06 10:32:27 [changch@hqdc034 ~]$ sudo service postgresql restart
<pre><code>2019-09-06 10:32:27 [changch@hqdc034 ~]$ sudo service postgresql restart
* Restarting PostgreSQL 10 database server [ OK ]
2019-09-06 10:33:37 [changch@hqdc034 ~]$
2019-09-06 10:33:37 [changch@hqdc034 ~]$ sudo netstat -antlp |grep 5432
</code></pre><p>而且在 syslog &amp; postgresql log 中看不到什麼異常，怪了！
而且再啟動一次就好了？</p>
<p>再來測試一次看看</p>
<pre tabindex="0"><code>2019-09-06 10:37:22 [changch@hqdc034 ~]$ sudo service postgresql stop
<pre><code>2019-09-06 10:37:22 [changch@hqdc034 ~]$ sudo service postgresql stop
* Stopping PostgreSQL 10 database server [ OK ]
2019-09-06 10:38:03 [changch@hqdc034 ~]$ sudo zfs list
NAME USED AVAIL REFER MOUNTPOINT
@@ -602,7 +602,7 @@ tcp6 0 0 :::5432 :::* LISTEN
2019-09-06 10:39:13 [changch@hqdc034 ~]$
</code></pre><p>這次就沒問題？看來是我第一次下指令的時候，不該用sudo netstat -antlp 去檢查？
anyway ，回到psql 來看看內容</p>
<pre tabindex="0"><code>postgres@hqdc034:~$ time psql -c 'select count(*) from test;'
<pre><code>postgres@hqdc034:~$ time psql -c 'select count(*) from test;'
count
----------
10000000
@@ -908,7 +908,7 @@ title="pinterest icon"></i>
</div>
<script src="https://h.cowbay.org/js/jquery.min.js"></script>
<script src="https://h.cowbay.org/js/jquerymigrate.js"></script>
<script src="https://h.cowbay.org/js/production.min.js?v=1634607506"></script>
<script src="https://h.cowbay.org/js/production.min.js?v=1635401692"></script>
</body>
</html>