Skip to content
23 August / Eric Chang / 筆記

# [筆記] 在Ubuntu 18.04 下 透過 pgbarman streaming backup 備份 postgresql 10/ backup postgresql 10 with pgbarman straming backup in ubuntu 18.04
some articles about job,food,passion sisters

很久以前就有看到這個用來備份postgresql 的 pgbarman

https://www.pgbarman.org/

前幾天老闆在slack 上面又提到，所以這次就花了點時間來玩玩看

不過呢，雖然有弄起來，但是還真不知道有些問題是怎麼解決的…

pgbarman 的備份有分兩種

streaming && rsync/SSH

原理就不說了，我一直沒搞懂 postgresql 的 streaming ..

依照官方網站上的說法，比較推薦 streaming 備份的方式

原因是，設定相對簡單，WTF !

    On a general basis, starting from Barman 2.0, backup over streaming replication is the recommended setup for PostgreSQL 9.4 or higher
    The reason why we recommend streaming backup is that, based on our experience, it is easier to setup than the traditional one

事實上呢，設定的確是很簡單，可是有個致命的缺點

    Because Barman transparently makes use of pg_basebackup, features such as incremental backup, parallel backup, deduplication, and network compression are currently not available. In this case, bandwidth limitation has some restrictions - compared to the traditional method via rsync.

如果要做差異/增量備份， streaming backup 不能做

所以每次備份都是完整備份，也因此，barman server 需要準備很大的硬碟空間

以我測試的資料庫來說，一次備份，目前是133G ，如果一天四次，保留七天

就需要 133 x 4 x 7 = 3724G

咦，這樣看起來，其實也還好啦 XDD

現在開始設定的部份

#### 設定 postgresql server

IP: 192.168.11.19 hostname: hqs019

##### 在postgresql 建立相關帳號

streaming backup 需要先在postgresql Server 上建立一個具有 superuser 權限的帳號

以及一個用來做replication 的資料庫帳號

這裡就簡單帶過

    sudo su - postgres
    psql

    create user barman with login superuser login password 'barmanpassword';
    CREATE ROLE stream_barman WITH REPLICATION PASSWORD 'password' LOGIN;

##### 鄉改 pg_hba.conf

然後修改 pg_hba.conf，加入底下兩行

    # for barman test
    host database_name barman 192.168.11.192/32 md5
    host replication stream_barman 192.168.11.192/32 md5

當然，如果不考慮安全性問題， md5 直接改成用 trust ，可以省去一些麻煩。

##### 修改 postgresql.conf

接著修改postgresql.conf

    ### for barman test
    max_wal_senders = 5
    max_replication_slots = 3
    wal_level = 'archive'
    archive_mode = on

重起 postgresql service

#### 設定barman server

IP: 192.168.11.192 hostname: barman

##### 安裝 barman

barman 在18.04 中，已經被放到標準repository 中

所以只要直接

    sudo apt install barman

就可以了

##### 設定 barman.conf

安裝完成後，在/etc/barman.d/ 底下會有兩個範例檔案

    streaming-server.conf-template
    ssh-server.conf-template

複製 streaming-server 檔案

    sudo cp /etc/barman.d/streaming-server.conf-template /etc/barman.d/hqs019.conf

內容如下

    [hqs019]
    description = "hqs019 "
    conninfo = host=192.168.11.19 user=barman dbname=database_name password=barmanpassword
    streaming_conninfo = host=192.168.11.19 user=stream_barman dbname=database_name password=password
    backup_method = postgres
    retention_policy_mode = auto
    streaming_archiver = on
    slot_name = barman

接著修改 /etc/barman.conf

    [barman]
    barman_user = barman
    configuration_files_directory = /etc/barman.d
    barman_home = /var/lib/barman
    log_file = /var/log/barman/barman.log
    log_level = DEBUG
    compression = gzip
    immediate_checkpoint = true
    basebackup_retry_times = 3
    basebackup_retry_sleep = 30
    last_backup_maximum_age = 1 DAYS

基本上這樣就設定完成了

##### 檢查設定

barman 有一些指令可以用來檢查目前的設定

barman show-server hqs019 可以看到所有的設定，這裡的 hqs019 跟 barman.d/hqs019.conf 裡面用”[ ]” 包起來的名稱要一致

    barman@barman:~$ barman show-server hqs019
    Server hqs019:
    	active: True
    	archiver: False
    	archiver_batch_size: 0
    	backup_directory: /var/lib/barman/hqs019
    	backup_method: postgres
    	backup_options: BackupOptions(['concurrent_backup'])
    	bandwidth_limit: None
    	barman_home: /var/lib/barman
    	barman_lock_directory: /var/lib/barman
    	basebackup_retry_sleep: 30
    	basebackup_retry_times: 3
    	basebackups_directory: /var/lib/barman/hqs019/base
    	check_timeout: 30
    	compression: gzip
    	config_file: /etc/postgresql/10/main/postgresql.conf
    	connection_error: None
    	conninfo: host=192.168.11.19 user=barman dbname=database_name password=barmanpassword
    	current_size: 142740768562
    	current_xlog: 0000000100000264000000BA
    	custom_compression_filter: None
    	custom_decompression_filter: None
    	data_directory: /database
    	description: hqs019
    	disabled: False
    	errors_directory: /var/lib/barman/hqs019/errors
    	hba_file: /etc/postgresql/10/main/pg_hba.conf
    	ident_file: /etc/postgresql/10/main/pg_ident.conf
    	immediate_checkpoint: True
    	incoming_wals_directory: /var/lib/barman/hqs019/incoming
    	is_in_recovery: False
    	is_superuser: True
    	last_backup_maximum_age: 1 day (WARNING! latest backup is No available backups old)
    	max_incoming_wals_queue: None
    	minimum_redundancy: 0
    	msg_list: []
    	name: hqs019
    	network_compression: False
    	parallel_jobs: 1
    	path_prefix: None
    	pg_basebackup_bwlimit: True
    	pg_basebackup_compatible: True
    	pg_basebackup_installed: True
    	pg_basebackup_path: /usr/bin/pg_basebackup
    	pg_basebackup_tbls_mapping: True
    	pg_basebackup_version: 10.10-0ubuntu0.18.04.1)
    	pg_receivexlog_compatible: True
    	pg_receivexlog_installed: True
    	pg_receivexlog_path: /usr/bin/pg_receivewal
    	pg_receivexlog_supports_slots: True
    	pg_receivexlog_synchronous: False
    	pg_receivexlog_version: 10.10-0ubuntu0.18.04.1)
    	pgespresso_installed: False
    	post_archive_retry_script: None
    	post_archive_script: None
    	post_backup_retry_script: None
    	post_backup_script: None
    	pre_archive_retry_script: None
    	pre_archive_script: None
    	pre_backup_retry_script: None
    	pre_backup_script: None
    	recovery_options: RecoveryOptions([])
    	replication_slot: Record(slot_name='barman', active=True, restart_lsn='264/BA000000')
    	replication_slot_support: True
    	retention_policy: None
    	retention_policy_mode: auto
    	reuse_backup: None
    	server_txt_version: 10.10
    	slot_name: barman
    	ssh_command: None
    	streaming: True
    	streaming_archiver: True
    	streaming_archiver_batch_size: 0
    	streaming_archiver_name: barman_receive_wal
    	streaming_backup_name: barman_streaming_backup
    	streaming_conninfo: host=192.168.11.19 user=stream_barman dbname=database_name password=password
    	streaming_supported: True
    	streaming_wals_directory: /var/lib/barman/hqs019/streaming
    	synchronous_standby_names: ['']
    	systemid: 6688476041000599317
    	tablespace_bandwidth_limit: None
    	timeline: 1
    	wal_level: replica
    	wal_retention_policy: main
    	wals_directory: /var/lib/barman/hqs019/wals
    	xlogpos: 264/BA000F08

然後用 barman check hqs019 來檢查config 有沒有問題

    barman@barman:~$ barman check hqs019
    Server hqs019:
    	PostgreSQL: OK
    	is_superuser: OK
    	PostgreSQL streaming: OK
    	wal_level: OK
    	replication slot: OK
    	directories: OK
    	retention policy settings: OK
    	backup maximum age: FAILED (interval provided: 1 day, latest backup age: No available backups)
    	compression settings: OK
    	failed backups: OK (there are 0 failed backups)
    	minimum redundancy requirements: OK (have 0 backups, expected at least 0)
    	pg_basebackup: OK
    	pg_basebackup compatible: OK
    	pg_basebackup supports tablespaces mapping: OK
    	pg_receivexlog: OK
    	pg_receivexlog compatible: OK
    	receive-wal running: OK
    	archiver errors: OK
    barman@barman:~$

那個backup maximum age FAILED 不用管他，因為都還沒跑過備份，這邊錯誤是正常的

其他都OK 的話，就可以開始備份了

barman backup hqs019

    barman@ubuntu:~$ barman backup hqs019
    Starting backup using postgres method for server hqs019 in /var/lib/barman/hqs019/base/20190823T082258
    Backup start at LSN: 264/A10001A8 (0000000100000264000000A1, 000001A8)
    Starting backup copy via pg_basebackup for 20190823T082258
    WARNING: pg_basebackup does not copy the PostgreSQL configuration files that reside outside PGDATA. Please manually backup the following files:
    	/etc/postgresql/10/main/postgresql.conf
    	/etc/postgresql/10/main/pg_hba.conf
    	/etc/postgresql/10/main/pg_ident.conf

    Copy done (time: 1 hour, 6 minutes, 28 seconds)
    Finalising the backup.
    Backup size: 133.0 GiB
    Backup end at LSN: 264/A3000060 (0000000100000264000000A3, 00000060)
    Backup completed (start time: 2019-08-23 08:22:58.116372, elapsed time: 1 hour, 6 minutes, 28 seconds)
    Processing xlog segments from streaming for hqs019
    	0000000100000264000000A2

    barman@ubuntu:~$

跑完可以用 barman list-backup hqs019 檢查

    barman@ubuntu:~$ barman list-backup hqs019
    hqs019 20190823T082258 - Thu Aug 22 17:29:26 2019 - Size: 133.0 GiB - WAL Size: 0 B (tablespaces: tablespace_a:/var/lib/postgresql/10/main/tablespace_A, tablespace_b:/var/lib/postgresql/10/main/tablespace_B)

要刪除的話，要加入 backupID

    barman@ubuntu:~$ barman delete hqs019 20190822T171355
    Deleting backup 20190822T171355 for server hqs019
    Delete associated WAL segments:
    	00000001000002640000009F
    	0000000100000264000000A0
    	0000000100000264000000A1
    Deleted backup 20190822T171355 (start time: Fri Aug 23 09:36:43 2019, elapsed time: 3 seconds)

restore 的部份，暫時沒有測試，我想應該是要找時間測試看看怎麼還原才對

不過呢，前面有提到，用streaming backup ，每一次備份都是完整備份，非常的佔用空間、時間、頻寬

所以還是要來試試看用rsync/SSH 備份的機制

Categories 筆記

Tags postgresql pgbarman

Written by:Eric Chang

塵世裡一個迷途小書僮

comments powered by Disqus