标签

PostgreSQL , 10.0 , 后台运行 , pg_background_launch , pg_background_result , pg_background_detach , pg_background

背景

当用户在管理数据库时，如果要在交互式界面跑一些QUERY，但是不知道QUERY要运行多久，担心网络问题或者其他问题导致终端断开，QUERY执行情况不明的话。就需要后台运行这个功能了。

后台运行在LINUX中也很常见，比如

nohup ls -la / >/tmp/result 2>&1 &

这样的话，即使断开会话也没关系，这条命令会在后台运行，并将标准输出存入/tmp/result，标准错误也重定向到标准输出。

对于PostgreSQL数据库，在10.0的这个特性出来之前，用户可以使用dblink的异步调用，达到类似的目的，但是不能断开会话，注意了。

dblink异步调用

使用dblink异步调用，可以达到后台运行的目的，但是别忘了，dblink连接是当前会话建立的，当前会话退出，连接也会退出。

postgres=# create extension dblink;  CREATE EXTENSION    创建测试表  postgres=# create table t(id int);  CREATE TABLE    建立连接  postgres=# select dblink_connect('digoal','host=127.0.0.1 port=1921 user=postgres dbname=postgres');   dblink_connect   ----------------   OK  (1 row)    开始事务（你也可以使用autocommit）  postgres=# select * from dblink_send_query('digoal', 'begin');   dblink_send_query   -------------------                   1  (1 row    获取异步调用结果  postgres=# select * from dblink_get_result('digoal') as t(res text);    res    -------   BEGIN  (1 row)    获取异步调用结果为0时，才能对这个连接进行下一次异步调用。  postgres=# select * from dblink_get_result('digoal') as t(res text);   res   -----  (0 rows)    异步插入数据  postgres=# select * from dblink_send_query('digoal', 'insert into t values (1)');   dblink_send_query   -------------------                   1  (1 row)    获取异步调用结果  postgres=# select * from dblink_get_result('digoal') as t(res text);      res       ------------   INSERT 0 1  (1 row)    postgres=# select * from dblink_get_result('digoal') as t(res text);   res   -----  (0 rows)    查看数据是否插入，因为异步事务没有提交，所以看不到数据  postgres=# select * from t;   id   ----  (0 rows)    提交异步事务  postgres=# select * from dblink_send_query('digoal', 'commit');   dblink_send_query   -------------------                   1  (1 row)    查看数据，有了  postgres=# select * from t;   id   ----    1  (1 row)

断开本地会话，异步会话也会断开，未提交的异步事务自动回滚。

postgres=# select dblink_connect('digoal','host=127.0.0.1 port=1921 user=postgres dbname=postgres');  -[ RECORD 1 ]--+---  dblink_connect | OK    postgres=# select * from dblink_send_query('digoal', 'begin');  -[ RECORD 1 ]-----+--  dblink_send_query | 1    postgres=# select * from dblink_get_result('digoal') as t(res text);  -[ RECORD 1 ]  res | BEGIN    postgres=# select * from dblink_get_result('digoal') as t(res text);  (0 rows)    postgres=# select * from dblink_send_query('digoal', 'insert into t values (2)');  -[ RECORD 1 ]-----+--  dblink_send_query | 1    退出当前会话  postgres=# \q    重新连接，异步会话已断开，并回滚。  postgres=# select * from t;   id   ----    1  (1 row)

使用dblink异步接口，可以完成一些后台运行的功能，但是比较繁琐，也不完美（比如当前会话不能退出）

PostgreSQL 10.0 新增了background session的功能，这个功能可以对标类似Oracle的自治事务的功能。（是plsql函数或存储过程迁移到PostgreSQL plpgsql的有利兼容性，此前需要使用dblink模拟自治事务）

基于background session，开发了一个后台运行的管理接口。可以方便的执行后台事务了。

PostgreSQL 10.0 background session(自治事务)功能

参考

PostgreSQL 10.0 后台运行接口功能

一开始的设计比较简单，提供了三个API函数

• pg_background_launch : 开启后台work进程与会话，执行用户提供的SQL，返回后台会话的PID    • pg_background_result   : 根据提供的PID，返回这个后台会话执行SQL的结果    • pg_background_detach : 根据提供的PID，返回这个后台会话执行SQL的结果，同时关闭这个后台进程。

最开始的讨论细节如下

Hi All,    I would like to take over pg_background patch and repost for  discussion and review.    Initially Robert Haas has share this for parallelism demonstration[1]  and abandoned later with  summary of open issue[2] with this pg_background patch need to be  fixed, most of them seems to be  addressed in core except handling of type exists without binary  send/recv functions and documentation.  I have added handling for types that don't have binary send/recv  functions in the attach patch and will  work on documentation at the end.    One concern with this patch is code duplication with  exec_simple_query(), we could  consider Jim Nasby’s patch[3] to overcome this,  but  certainly we  will end up by complicating  exec_simple_query() to make pg_background happy.    As discussed previously[1] pg_background is a contrib module that lets  you launch arbitrary command in a background worker.    • VACUUM in background  • Autonomous transaction implementation better than dblink way (i.e.  no separate authentication required).  • Allows to perform task like CREATE INDEX CONCURRENTLY from a  procedural language.    This module comes with following SQL APIs:    • pg_background_launch : This API takes SQL command, which user wants  to execute, and size of queue buffer.    This function returns the process id of background worker.  • pg_background_result   : This API takes the process id as input  parameter and returns the result of command    executed thought the background worker.  • pg_background_detach : This API takes the process id and detach the  background process which is waiting for   user to read its results.      Here's an example of running vacuum and then fetching the results.  Notice that the  notices from the original session are propagated to our session; if an  error had occurred,  it would be re-thrown locally when we try to read the results.    postgres=# create table foo (a int);  CREATE TABLE  postgres=# insert into foo values(generate_series(1,5));  INSERT 0 5    postgres=# select pg_background_launch('vacuum verbose foo');  pg_background_launch  ----------------------                65427  (1 row)    postgres=# select * from pg_background_result(65427) as (x text);  INFO:  vacuuming "public.foo"  INFO:  "foo": found 0 removable, 5 nonremovable row versions in 1 out of 1 pages  DETAIL:  0 dead row versions cannot be removed yet.  There were 0 unused item pointers.  Skipped 0 pages due to buffer pins.  0 pages are entirely empty.  CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.   x  --------  VACUUM  (1 row)      Thanks to Vibhor kumar, Rushabh Lathia and Robert Haas for feedback.    Please let me know your thoughts, and thanks for reading.    [1]. https://www.postgresql.org/message-id/CA%2BTgmoam66dTzCP8N2cRcS6S6dBMFX%2BJMba%2BmDf68H%3DKAkNjPQ%40mail.gmail.com  [2]. https://www.postgresql.org/message-id/CA%2BTgmobPiT_3Qgjeh3_v%2B8Cq2nMczkPyAYernF_7_W9a-6T1PA%40mail.gmail.com  [3]. https://www.postgresql.org/message-id/54541779.1010906%40BlueTreble.com    Regards,  Amul

社区讨论后，这个架构改成了这样的，架构更优雅一些。

• pg_background_launch : 这个接口只是用来fork一个后台进程，并返回PID    • pg_background_run : 根据提供的PID，让这个后台进程执行提供的SQL。    • pg_background_result : 根据提供的PID，获取执行SQL的结果。    • pg_background_detach : 关闭后台进程与会话。

讨论细节如下

Hi all,    As we have discussed previously, we need to rework this patch as a client of  Peter Eisentraut's background sessions code[1].    Attaching trial version patch to discussed possible design and api.  We could have following APIs :    • pg_background_launch : This function start and stores new background  session, and returns the process id of background worker.    • pg_background_run : This API takes the process id and SQL command as  input parameter. Using this process id, stored worker's session is  retrieved and give SQL command is executed under it.    • pg_background_result : This API takes the process id as input  parameter and returns the result of command executed thought the  background worker session.  Same as it was before but now result can  be fetch in LIFO order i.e. result of last executed query using  pg_background_run will be fetched first.    • pg_background_detach : This API takes the process id and detach the  background process. Stored worker's session is not dropped until this  called.    • TBC : API to discard result of last query or discard altogether?    • TBC : How about having one more api to see all existing sessions ?      Kindly share your thoughts/suggestions.  Note that attach patch is WIP  version, code, comments & behaviour could be vague.    ------------------  Quick demo:  ------------------  Apply attach patch to the top of Peter Eisentraut's  0001-Add-background-sessions.patch[1]    postgres=# select pg_background_launch();   pg_background_launch  ----------------------                  21004  (1 row)    postgres=# select pg_background_run(21004, 'vacuum verbose foo');   pg_background_run  -------------------    (1 row)    postgres=# select * from pg_background_result(21004) as (x text);  INFO:  vacuuming "public.foo"  INFO:  "foo": found 0 removable, 5 nonremovable row versions in 1 out of 1 pages  DETAIL:  0 dead row versions cannot be removed yet.  There were 0 unused item pointers.  Skipped 0 pages due to buffer pins.  0 pages are entirely empty.  CPU: user: 0.00 s, system: 0.00 s, elapsed: 0.00 s.     x  --------   VACUUM  (1 row)    postgres=# select pg_background_run(21004, 'select * from foo');   pg_background_run  -------------------    (1 row)    postgres=# select * from pg_background_result(21004) as (x int);   x  ---   1   2   3   4   5  (5 rows)    postgres=# select pg_background_detach(21004);   pg_background_detach  ----------------------    (1 row)      References :  [1] https://www.postgresql.org/message-id/e1c2d331-ee6a-432d-e9f5-dcf85cffaf29%402ndquadrant.com.      Regards,  Amul Sul

后面的讨论又改成了这样，还是朝着优雅的方向在改进

The following review has been posted through the commitfest application:  make installcheck-world:  tested, passed  Implements feature:       tested, passed  Spec compliant:           tested, passed  Documentation:            tested, failed    I’ll summarize here my thoughts as a reviewer on the current state of the pg_background:  1. Current version of a code [1] is fine, from my point of view. I have no suggestions on improving it. There is no documentation, but code is commented.  2. Patch is dependent on background sessions from the same commitfest.  3. There can exist more features, but for v1 there is surely enough features.  4. There is some controversy on where implemented feature shall be: in separate extension (as in this patch), in db_link, in some PL API, in FDW or somewhere else.   I think that new extension is an appropriate place for the feature. But I’m not certain.  Summarizing these points, appropriate statuses of the patch are ‘Ready for committer’ or ‘Rejected’.   Between these two I choose ‘Ready for committer’, I think patch is committable (after bg sessions).    Best regards, Andrey Borodin.    The new status of this patch is: Ready for Committer

这个patch在commit前，还有一些变数，比如可能将这个功能合并到dblink里面。而不是新开一个extension插件.

这个patch的讨论，详见邮件组，本文末尾URL。

PostgreSQL社区的作风非常严谨，一个patch可能在邮件组中讨论几个月甚至几年，根据大家的意见反复的修正，patch合并到master已经非常成熟，所以PostgreSQL的稳定性也是远近闻名的。

参考