前往小程序,Get更优阅读体验!
立即前往
首页
学习
活动
专区
圈层
工具
发布
首页
学习
活动
专区
圈层
工具
MCP广场
社区首页 >专栏 >利用OGG实现Oracle到Kafka到Greenplum的增量数据同步

利用OGG实现Oracle到Kafka到Greenplum的增量数据同步

作者头像
数据和云
发布于 2021-03-09 07:40:23
发布于 2021-03-09 07:40:23
1.9K00
代码可运行
举报
文章被收录于专栏:数据和云数据和云
运行总次数:0
代码可运行

墨墨导读:本文来自墨天轮用户 肖杰 的投稿,介绍用OGG实现Oracle到Kafka到Greenplum的增量数据同步的全过程。

墨天轮主页:https://www.modb.pro/u/6722

背景

在大数据库时代,数据经常需要在不同的数据库之间流动、整合,并要求具有一定的实时性,传统的通过脚本定时,批量同步的方式根本无法满足需求。

本文基于Oracle OGG,Kafka消息队列实现Oracle到Greenplum之间的准实时同步(实测延时在ms级别)。

一、环境准备

版本

OGG版本

IP

源端

oracle 12.2.0.1

123012_fbo_ggs_Linux_x64_shiphome.zip

192.168.11.151

目标端

kafka 2.12

OGG_BigData_Linux_x64_12.3.2.1.1.zip

192.168.11.165/4/3

二、ORACLE,GREENPLUM,OGG安装

此过程所用方法比较通用,此处不再赘述。

三、源库配置

源库需要开启归档,配置ogg用户,表空间及相关参数等,此处不再详细赘述

四、源端OGG配置

1. 进程配置 mgr.prm

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
PORT 7809
代码语言:javascript
代码运行次数:0
运行
复制

extn.prm

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
extract extn
userid C##GGADMIN@ora12c,password GGadmin_123
DISCARDFILE ./dirrpt/extn.dsc, APPEND, MEGABYTES 1024
EXTTRAIL  ./dirdat/na
GETUPDATEBEFORES
NOCOMPRESSUPDATES
NOCOMPRESSDELETES
DYNAMICRESOLUTION
--源库是一个CDB,本次测试抽取pdb_test下面的两张表
table pdb_test.test.ccps_traderecord;
table pdb_test.test.ccps_unnormal_traderecord;
代码语言:javascript
代码运行次数:0
运行
复制

dpen.prm

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
extract dpen
RMTHOST 192.168.11.165, MGRPORT 7809
RMTTRAIL ./dirdat/na
DYNAMICRESOLUTION
TABLE pdb_test.test.*;
代码语言:javascript
代码运行次数:0
运行
复制

2. 创建进程

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
add extract extn, TRANLOG, BEGIN NOW
add exttrail ./dirdat/na, EXTRACT extn, MEGABYTES 200

ADD EXTRACT dpen, EXTTRAILSOURCE ./dirdat/na
ADD RMTTRAIL ./dirdat/na, EXTRACT dpen, MEGABYTES 200
代码语言:javascript
代码运行次数:0
运行
复制

五、目标端OGG配置

1. 配置JDBC Hander参数

示例目录下面有配置模板,复制过来即可 cp /home/oracle/ogg/AdapterExamples/big-data/kafka/* /home/oracle/ogg/dirprm

custom_kafka_producer.properties:

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
bootstrap.servers=localhost:9092
acks=1
reconnect.backoff.ms=1000

value.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
key.serializer=org.apache.kafka.common.serialization.ByteArraySerializer
# 100KB per partition
batch.size=16384
linger.ms=0
max.request.size = 5024000
send.buffer.bytes = 5024000
代码语言:javascript
代码运行次数:0
运行
复制

kafka.props

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
gg.handlerlist = kafkahandler
gg.handler.kafkahandler.type=kafka
gg.handler.kafkahandler.KafkaProducerConfigFile=custom_kafka_producer.properties
#The following resolves the topic name using the short table name
gg.handler.kafkahandler.topicMappingTemplate=oggtopic
#The following selects the message key using the concatenated primary keys
#gg.handler.kafkahandler.keyMappingTemplate=${primaryKeys}
gg.handler.kafkahandler.format=json
gg.handler.kafkahandler.SchemaTopicName=mySchemaTopic
gg.handler.kafkahandler.BlockingSend =false
gg.handler.kafkahandler.includeTokens=false
gg.handler.kafkahandler.mode=tx

goldengate.userexit.writers=javawriter
javawriter.stats.display=TRUE
javawriter.stats.full=TRUE

gg.log=log4j
gg.log.level=INFO

gg.report.time=30sec

#Sample gg.classpath for Apache Kafka
gg.classpath=dirprm/:/home/oracle/kafka/libs/*
#Sample gg.classpath for HDP
#gg.classpath=/etc/kafka/conf:/usr/hdp/current/kafka-broker/libs/*

javawriter.bootoptions=-Xmx512m -Xms512m -Djava.class.path=ggjava/ggjava.jar
代码语言:javascript
代码运行次数:0
运行
复制

2. 进程配置

mgr.prm

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
port 7809
代码语言:javascript
代码运行次数:0
运行
复制

repn.prm

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
replicat repn
getenv(JAVA_HOME)
getenv(CLASSPATH)
getenv(PATH)
getenv(LD_LIBRARY_PATH)
SETENV (NLS_LANG="AMERICAN_AMERICA.AL32UTF8")
TARGETDB LIBFILE libggjava.so SET property=./dirprm/kafka.props
SOURCEDEFS /home/oracle/ogg/dirdef/source.def
REPORTCOUNT EVERY 1 MINUTES, RATE
--GROUPTRANSOPS 10000
MAP pdb_test.test.ccps_traderecord, TARGET ccps.ccps_traderecord;
MAP pdb_test.test.ccps_unnormal_traderecord, TARGET ccps.ccps_unnormal_traderecord;
代码语言:javascript
代码运行次数:0
运行
复制

3. 创建进程

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
add replicat repn, exttrail ./dirdat/na
代码语言:javascript
代码运行次数:0
运行
复制

六、目标端kafka安装配置

1. 安装

解压即可(此版本kafka本身已经集成zookeeper,不需要单独安装。)

2. 配置

因为是测试环境,此处列出的都是一些关键性参数,至于其他性能相关参数根据自己需要调整即可。

consumer.properties

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
bootstrap.servers=localhost:9092
代码语言:javascript
代码运行次数:0
运行
复制

server.properties

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
listeners=PLAINTEXT://localhost:9092
zookeeper.connect=localhost:2181
代码语言:javascript
代码运行次数:0
运行
复制

zookeeper.properties

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
clientPort=2181
代码语言:javascript
代码运行次数:0
运行
复制

3. 启动zookeeper

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
[oracle@gpmaster kafka]$ bin/zookeeper-server-start.sh -daemon config/zookeeper.properties
[oracle@gpmaster kafka]$ jps
18001 Jps
2505 QuorumPeerMain
代码语言:javascript
代码运行次数:0
运行
复制

4. 启动kafka

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
[oracle@gpmaster kafka]$ bin/kafka-server-start.sh config/server.properties 
[oracle@gpmaster kafka]$ jps
18016 Kafka
2505 QuorumPeerMain
18458 Jps
代码语言:javascript
代码运行次数:0
运行
复制

5. kafka常用命令

显示所有topic(其中oggtopic就是用于此次测试的topic,kafka.props中配置的名字):

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
[oracle@gpmaster bin]$ kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
mySchemaTopic
oggtest
oggtopic
test

生产消息:

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
[oracle@gpmaster bin]$ kafka-console-producer.sh --broker-list localhost:9092 --topic test
>test kafka!!!!
>

消费消息:

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
[oracle@gpmaster bin]$ kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test     
test kafka!!!!

七、编写代码将数据从kafka加载到GP

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
from kafka import KafkaConsumer
import json
import psycopg2
import time
import re

dbconn = psycopg2.connect(dbname = "devin",user = "ccps",password = "xjlove1224",host = "192.168.11.165",port = "5432")

consumer = KafkaConsumer('oggtopic',group_id = "oggtopic",auto_offset_reset='earliest',bootstrap_servers=['localhost:9092'])
for msg in consumer:
    message = msg.value
    mesg = message.decode('utf-8')
    msglist = re.findall(r'(.*?}})',mesg)  --切分
代码语言:javascript
代码运行次数:0
运行
复制

代码细节略,需要注意NULL,各种特殊字符以及异常的处理。

八、测试

测试环境,目标端没有做初始化,运行结果如下:

抽取到kafka的数据,json格式:

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
/*
* 提示:该行代码过长,系统自动注释不进行高亮。一键复制会移除系统注释 
* {"table":"CCPS.CCPS_UNNORMAL_TRADERECORD","op_type":"I","op_ts":"2020-08-18 18:15:48.048933","current_ts":"2020-08-18T18:15:54.146000","pos":"00000000000026312368","after":{"UTR_ID":923628,"UTR_NO":"200818181648559171322","UTR_MER_NO":191713,"UTR_GW_NO":19171301,"UTR_MER_ORDERNO":"1597745803378","UTR_CURRENCY":"USD","UTR_AMOUNT":"0.01","UTR_BANK_CODE":null,"UTR_CHA_CODE":null,"UTR_RETURNURL":null,"UTR_WEBSITE":"http://192.168.11.98:7105/Test/jsp/testPayTwo.jsp","UTR_SUBMITURL":"http://192.168.11.98:7105/PaymentGateway/gateway/directservice/pay","UTR_PAYSTARTTIME":"2020-08-18 18:16:48.233971000","UTR_PAYTIME":null,"UTR_PAYENDTIME":null,"UTR_ERRORCODE":"50000","UTR_ERRORREASON_OUT":"Unpaid","UTR_ERRORREASON_IN":"Unpaid","UTR_MER_REMARK":"2fang","UTR_RISKINFO":null,"UTR_SETSCORE":null,"UTR_TOTALSCORE":null,"UTR_PASSRISKINFO":null,"UTR_PM_ID":1,"UTR_CARDNO_ENCRYPT":null,"UTR_NOTICEURL":null,"UTR_BILL_ADDRESS_WHITE":null,"UTR_BILL_ADDRESS_EXTRA1_WHITE":null,"UTR_LOGO_URL":null,"UTR_LANGUAGE":null,"UTR_PAY_EXPIRATIONTIME":null,"UTR_MCC":null,"UTR_PM_NAME":"Credit Card","UTR_TRADE_BANK_NAME":null,"UTR_EXPIRATION_TIME":null}}{"table":"CCPS.CCPS_UNNORMAL_TRADERECORD","op_type":"U","op_ts":"2020-08-18 18:15:48.048933","current_ts":"2020-08-18T18:15:54.167000","pos":"00000000000026316287","before":{},"after":{"UTR_NO":"200818181648559171322","UTR_CARDNO_ENCRYPT":"GuL6A5JN4rwAaHb8fPSAMdhxRA9CL1RVIjWwP81YujHEvXuz0YnmBssWqslPmsAgsuPeKBdQarDb4B23oy0C6wMJYner9LKmRVbuNfuGummiWnRmHAM9Dn2dJKcujEPProxYeBWUMbEhk9V54Q018cML1bcf3fvJjKL/vt4xBbk="}}{"table":"CCPS.CCPS_UNNORMAL_TRADERECORD","op_type":"D","op_ts":"2020-08-18 18:15:48.048933","current_ts":"2020-08-18T18:15:54.167001","pos":"00000000000026316959","before":{"UTR_ID":923628,"UTR_NO":"200818181648559171322","UTR_MER_NO":191713,"UTR_GW_NO":19171301,"UTR_MER_ORDERNO":"1597745803378","UTR_CURRENCY":"USD","UTR_AMOUNT":"0.01","UTR_BANK_CODE":null,"UTR_CHA_CODE":null,"UTR_RETURNURL":null,"UTR_WEBSITE":"http://192.168.11.98:7105/Test/jsp/testPayTwo.jsp","UTR_SUBMITURL":"http://192.168.11.98:7105/PaymentGateway/gateway/directservice/pay","UTR_PAYSTARTTIME":"2020-08-18 18:16:48.233971000","UTR_PAYTIME":null,"UTR_PAYENDTIME":null,"UTR_ERRORCODE":"50000","UTR_ERRORREASON_OUT":"Unpaid","UTR_ERRORREASON_IN":"Unpaid","UTR_MER_REMARK":"2fang","UTR_RISKINFO":null,"UTR_SETSCORE":null,"UTR_TOTALSCORE":null,"UTR_PASSRISKINFO":null,"UTR_PM_ID":1,"UTR_CARDNO_ENCRYPT":"GuL6A5JN4rwAaHb8fPSAMdhxRA9CL1RVIjWwP81YujHEvXuz0YnmBssWqslPmsAgsuPeKBdQarDb4B23oy0C6wMJYner9LKmRVbuNfuGummiWnRmHAM9Dn2dJKcujEPProxYeBWUMbEhk9V54Q018cML1bcf3fvJjKL/vt4xBbk=","UTR_NOTICEURL":null,"UTR_BILL_ADDRESS_WHITE":null,"UTR_BILL_ADDRESS_EXTRA1_WHITE":null,"UTR_LOGO_URL":null,"UTR_LANGUAGE":null,"UTR_PAY_EXPIRATIONTIME":null,"UTR_MCC":null,"UTR_PM_NAME":"Credit Card","UTR_TRADE_BANK_NAME":null,"UTR_EXPIRATION_TIME":null}}{"table":"CCPS.CCPS_TRADERECORD","op_type":"I","op_ts":"2020-08-18 18:15:48.048933","current_ts":"2020-08-18T18:15:54.168000","pos":"00000000000026318029","after":{"TR_ID":787160,"TR_NO":"200818181648559171322","TR_MER_ORDERNO":"1597745803378","TR_MER_NO":191713,"TR_GW_NO":19171301,"TR_CURRENCY":"USD","TR_AMOUNT":0.01,"TR_STATUS":-1,"TR_TRADE_RATE":0.0100,"TR_SPP_CURRENCY":"USD","TR_SPP":0,"TR_AGENT_NO":0,"TR_AGENT_RATE":null,"TR_AGENT_SPP_CURRENCY":null,"TR_AGENT_SPP":null,"TR_RESEVER_RATE":0.1000,"TR_RATE_VALUE":1,"TR_BANKCURRENCY":"USD","TR_BANKAMOUT":0.01,"TR_BANK_CODE":"OPBank","TR_CHA_CODE":121,"TR_ISDELAY":0,"TR_CHA_RATE":0.0100,"TR_BANK_SPP_CURRENCY":"USD","TR_BANK_SPP":1.00,"TR_CHA_SETT_BANK":null,"TR_BANKORDERNO":null,"TR_BANKRETURNCODE":null,"TR_BANKINFO":null,"TR_PAYSTARTTIME":"2020-08-18 18:16:48.139000000","TR_PAYENDTIME":null,"TR_DATETIME":"2020-08-18 18:16:48.346618000","TR_BANKDATETIME":null,"TR_RETURNURL":null,"TR_WEBSITE":"http://192.168.11.98:7105/Test/jsp/testPayTwo.jsp","TR_SUBMITURL":"http://192.168.11.98:7105/PaymentGateway/gateway/directservice/pay","TR_CHECKED":0,"TR_CHECKDATETIME":null,"TR_REFUNDMENT":0,"TR_PROTEST":0,"TR_CONGEAL":0,"TR_DELIVERY":null,"TR_REFUNDMENT_AMOUNT":0,"TR_PROTEST_AMOUNT":0,"TR_EXP_AMOUNT":0,"TR_CONGEAL_AMOUNT":0,"TR_UNSETT_AMOUNT":null,"TR_TS_ID":0,"TR_TS_STATUS":0,"TR_RS_ID":0,"TR_RS_STATUS":0,"TR_AS_ID":0,"TR_AS_STATUS":0,"TR_QUERYNO":null,"TR_AUTHORIZELD":null,"TR_BATCHNO":null,"TR_TERMINALNO":null,"TR_ISDCC":0,"TR_TS_BATCH":0,"TR_ISTSCHECK":0,"TR_RS_BATCH":0,"TR_ISRSCHECK":0,"TR_ISDAYCHECK":0,"TR_ISEXPLAIN":-2,"TR_ISFRAUD":0,"TR_ISCOMPLAIN":0,"TR_INF_TYPE":2,"TR_ISLOCK":0,"TR_FEE_FAIL_MER":0,"TR_FEE_SUCCESS_MER":0,"TR_FEE_SUCCESS_AFTER_MER":0,"TR_IS_BACK_MER":0,"TR_IS_BACK_AFTER_MER":0,"TR_FEE_FAIL_AGENT":0,"TR_FEE_SUCCESS_AGENT":0,"TR_FEE_SUCCESS_AFTER_AGENT":0,"TR_IS_BACK_AGENT":0,"TR_IS_BACK_AFTER_AGENT":0,"TR_FEE_FAIL_CHA":1,"TR_FEE_SUCCESS_CHA":1,"TR_FEE_SUCCESS_AFTER_CHA":1,"TR_IS_BACK_CHA":1,"TR_IS_BACK_AFTER_CHA":1,"TR_REMARK":"2fang","TR_BANKTRADETIME":null,"TR_CARDTYPE":1,"TR_REFERENCE":"200818181648559171322","TR_IS_REPAY":0,"TR_RISKINFO":null,"TR_SETSCORE":0,"TR_TOTALSCORE":98.00,"TR_PASSRISKINFO":null,"TR_PM_ID":1,"TR_AUTH_TYPE":0,"TR_SEND_CR_FLAG":null,"TR_SEND_MER_FLAG":null,"TR_REFCHAPRO_FLAG":null,"TR_CLEAR_AMOUNT":null,"TR_CLEAR_TIME":null,"TR_CLEAR_STATUS":null,"TR_BANK_CHARGES":null,"TR_CLEAR_CURRENCY":null,"TR_AUTH_OP_TIME":null,"TR_DEBITTIME":"2020-08-18 18:16:48.346618000","TR_SETT_RATE":1,"TR_SETT_AMOUNT":0.01,"TR_SETT_CURRENCY":"USD","TR_TRANRETURNCODE":null,"TR_TRANINFO":null,"TR_WHETHER_3D":"0","TR_ECI_VALUE":null,"TR_ISAUTH":0,"TR_SETTLEMENT":0,"TR_NOTICEURL":null,"TR_NOTICESTATUS":null,"TR_CHARESERVERATE":0.0100,"TR_REFUSEAMOUNT":0,"TR_REFOUNDAMOUNT":0,"TR_BAILAMOUNT":null,"TR_IS_MER_REFOUND":0,"TR_IS_MER_REFUSE":0,"TR_SETT_SPP":0,"TR_BILL_ADDRESS":"OPBANK 账单地址","TR_PAY_SOLUTIONS":null,"TR_PAY_INSTALLCOUNT":null,"TR_PAY_INSTALLRATE":null,"TR_SPP_RATE":1,"TR_RATE_INSTALLRATE":0,"TR_MER_LOGOURL":null,"TR_CLEAR_BATCHNO":null,"TR_CLEAR_REMARK":null,"TR_IS_MER_REFOUND_PART":0,"TR_BILL_ADDRESS_WHITE":null,"TR_BILL_ADDRESS_EXTRA1_WHITE":null,"TR_PRIOR_CHANNEL":0,"TR_SETTLEMENT_BANK":null,"TR_SUB_BATCHNO_BANK":null,"TR_SUB_BANK_NO":null,"TR_BANK_REMARK":null,"TR_CONTRACTTYPE":302,"TR_SETTLEMENTBODY":8,"TR_CHANNELBODY":2,"TR_SETTLEMENTAMONT":null,"TR_TSINGLEFEE":null,"TR_TREFUNDFEE":null,"TR_TRATE":null,"TR_TRATERETURN":null,"TR_TAXATION":null,"TR_MIN_TRADE_FEE":0,"TR_BANK_RATE_HKD":7.75081,"TR_SETT_RATE_HKD":7.75081,"TR_RETURN_AVS":null,"TR_RETURN_CVV":null,"TR_TYPE":1,"TR_SETT_STATUS":null,"TR_REFUND_LOCK":null,"TR_MCC":5722,"TR_UPDATE_ORDER_STATUS":0,"TR_MANUAL_LIQUIDATION_REMARK":null,"TR_OP_ENTITY":1,"TR_TRADE_BANK_NAME":null,"TR_CLEAR_RATE":null,"TR_INTERCHANGE_FEE":null,"TR_SCHEME_FEE":null,"TR_ACQUIRER_FEE":null,"TR_CLEAR_TRADE_RATE":null,"TR_CROSS_BORDER_FEE":null,"TR_AVS_OUT_RESULT":null,"TR_CAVV":null,"TR_BANK_PID":null,"TR_EXPIRATION_TIME":null,"TR_TCHARGEBACK_FEE":null,"TR_TCONGEAL_FEE":null,"TR_FXAJD_FEE":null,"TR_TOTHER_FEE":null}}{"table":"CCPS.CCPS_TRADERECORD","op_type":"U","op_ts":"2020-08-18 18:15:48.048933","current_ts":"2020-08-18T18:15:54.168001","pos":"00000000000026320941","before":{},"after":{"TR_NO":"200818181648559171322","TR_STATUS":1,"TR_BANKORDERNO":null,"TR_BANKRETURNCODE":"80000","TR_BANKINFO":"Transaction Approved1","TR_BANKDATETIME":"2020-08-18 18:16:48.360706000","TR_CHECKED":0,"TR_CHECKDATETIME":"2020-08-18 18:16:48.000000000","TR_QUERYNO":null,"TR_AUTHORIZELD":null,"TR_BATCHNO":null,"TR_TERMINALNO":null,"TR_AUTH_TYPE":0,"TR_DEBITTIME":"2020-08-18 18:16:48.360706000","TR_TRANRETURNCODE":"10000","TR_TRANINFO":"Transaction Approved","TR_WHETHER_3D":"0","TR_ECI_VALUE":null,"TR_NOTICESTATUS":2,"TR_PAY_SOLUTIONS":"Transaction Approved3","TR_BANK_REMARK":";;","TR_RETURN_AVS":null,"TR_RETURN_CVV":null,"TR_UPDATE_ORDER_STATUS":0}}
*/
{"table":"CCPS.CCPS_TRADERECORD","op_type":"U","op_ts":"2020-08-18 18:16:01.042146","current_ts":"2020-08-18T18:16:06.549000","pos":"00000000000026321695","before":{},"after":{"TR_NO":"160627142027150120836","TR_RS_ID":1000779,"TR_RS_STATUS":2,"TR_UPDATE_ORDER_STATUS":0}}
代码语言:javascript
代码运行次数:0
运行
复制

python解析后的SQL:

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
insert into CCPS_UNNORMAL_TRADERECORD(UTR_ID, UTR_NO, UTR_MER_NO, UTR_GW_NO, UTR_MER_ORDERNO, UTR_CURRENCY, UTR_AMOUNT, UTR_BANK_CODE, UTR_CHA_CODE, UTR_RETURNURL, UTR_WEBSITE, UTR_SUBMITURL, UTR_PAYSTARTTIME, UTR_PAYTIME, UTR_PAYENDTIME, UTR_ERRORCODE, UTR_ERRORREASON_OUT, UTR_ERRORREASON_IN, UTR_MER_REMARK, UTR_RISKINFO, UTR_SETSCORE, UTR_TOTALSCORE, UTR_PASSRISKINFO, UTR_PM_ID, UTR_CARDNO_ENCRYPT, UTR_NOTICEURL, UTR_BILL_ADDRESS_WHITE, UTR_BILL_ADDRESS_EXTRA1_WHITE, UTR_LOGO_URL, UTR_LANGUAGE, UTR_PAY_EXPIRATIONTIME, UTR_MCC, UTR_PM_NAME, UTR_TRADE_BANK_NAME, UTR_EXPIRATION_TIME) values (923628, '200818181648559171322', 191713, 19171301, '1597745803378', 'USD', '0.01', NULL, NULL, NULL, 'http://192.168.11.98:7105/Test/jsp/testPayTwo.jsp', 'http://192.168.11.98:7105/PaymentGateway/gateway/directservice/pay', '2020-08-18 18:16:48.233971000', NULL, NULL, '50000', 'Unpaid', 'Unpaid', '2fang', NULL, NULL, NULL, NULL, 1, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, 'Credit Card', NULL, NULL);
update CCPS_UNNORMAL_TRADERECORD set UTR_CARDNO_ENCRYPT = 'GuL6A5JN4rwAaHb8fPSAMdhxRA9CL1RVIjWwP81YujHEvXuz0YnmBssWqslPmsAgsuPeKBdQarDb4B23oy0C6wMJYner9LKmRVbuNfuGummiWnRmHAM9Dn2dJKcujEPProxYeBWUMbEhk9V54Q018cML1bcf3fvJjKL/vt4xBbk=' where UTR_NO = '200818181648559171322'
delete from CCPS_UNNORMAL_TRADERECORD where UTR_ID = 923628
insert into CCPS_TRADERECORD(TR_ID, TR_NO, TR_MER_ORDERNO, TR_MER_NO, TR_GW_NO, TR_CURRENCY, TR_AMOUNT, TR_STATUS, TR_TRADE_RATE, TR_SPP_CURRENCY, TR_SPP, TR_AGENT_NO, TR_AGENT_RATE, TR_AGENT_SPP_CURRENCY, TR_AGENT_SPP, TR_RESEVER_RATE, TR_RATE_VALUE, TR_BANKCURRENCY, TR_BANKAMOUT, TR_BANK_CODE, TR_CHA_CODE, TR_ISDELAY, TR_CHA_RATE, TR_BANK_SPP_CURRENCY, TR_BANK_SPP, TR_CHA_SETT_BANK, TR_BANKORDERNO, TR_BANKRETURNCODE, TR_BANKINFO, TR_PAYSTARTTIME, TR_PAYENDTIME, TR_DATETIME, TR_BANKDATETIME, TR_RETURNURL, TR_WEBSITE, TR_SUBMITURL, TR_CHECKED, TR_CHECKDATETIME, TR_REFUNDMENT, TR_PROTEST, TR_CONGEAL, TR_DELIVERY, TR_REFUNDMENT_AMOUNT, TR_PROTEST_AMOUNT, TR_EXP_AMOUNT, TR_CONGEAL_AMOUNT, TR_UNSETT_AMOUNT, TR_TS_ID, TR_TS_STATUS, TR_RS_ID, TR_RS_STATUS, TR_AS_ID, TR_AS_STATUS, TR_QUERYNO, TR_AUTHORIZELD, TR_BATCHNO, TR_TERMINALNO, TR_ISDCC, TR_TS_BATCH, TR_ISTSCHECK, TR_RS_BATCH, TR_ISRSCHECK, TR_ISDAYCHECK, TR_ISEXPLAIN, TR_ISFRAUD, TR_ISCOMPLAIN, TR_INF_TYPE, TR_ISLOCK, TR_FEE_FAIL_MER, TR_FEE_SUCCESS_MER, TR_FEE_SUCCESS_AFTER_MER, TR_IS_BACK_MER, TR_IS_BACK_AFTER_MER, TR_FEE_FAIL_AGENT, TR_FEE_SUCCESS_AGENT, TR_FEE_SUCCESS_AFTER_AGENT, TR_IS_BACK_AGENT, TR_IS_BACK_AFTER_AGENT, TR_FEE_FAIL_CHA, TR_FEE_SUCCESS_CHA, TR_FEE_SUCCESS_AFTER_CHA, TR_IS_BACK_CHA, TR_IS_BACK_AFTER_CHA, TR_REMARK, TR_BANKTRADETIME, TR_CARDTYPE, TR_REFERENCE, TR_IS_REPAY, TR_RISKINFO, TR_SETSCORE, TR_TOTALSCORE, TR_PASSRISKINFO, TR_PM_ID, TR_AUTH_TYPE, TR_SEND_CR_FLAG, TR_SEND_MER_FLAG, TR_REFCHAPRO_FLAG, TR_CLEAR_AMOUNT, TR_CLEAR_TIME, TR_CLEAR_STATUS, TR_BANK_CHARGES, TR_CLEAR_CURRENCY, TR_AUTH_OP_TIME, TR_DEBITTIME, TR_SETT_RATE, TR_SETT_AMOUNT, TR_SETT_CURRENCY, TR_TRANRETURNCODE, TR_TRANINFO, TR_WHETHER_3D, TR_ECI_VALUE, TR_ISAUTH, TR_SETTLEMENT, TR_NOTICEURL, TR_NOTICESTATUS, TR_CHARESERVERATE, TR_REFUSEAMOUNT, TR_REFOUNDAMOUNT, TR_BAILAMOUNT, TR_IS_MER_REFOUND, TR_IS_MER_REFUSE, TR_SETT_SPP, TR_BILL_ADDRESS, TR_PAY_SOLUTIONS, TR_PAY_INSTALLCOUNT, TR_PAY_INSTALLRATE, TR_SPP_RATE, TR_RATE_INSTALLRATE, TR_MER_LOGOURL, TR_CLEAR_BATCHNO, TR_CLEAR_REMARK, TR_IS_MER_REFOUND_PART, TR_BILL_ADDRESS_WHITE, TR_BILL_ADDRESS_EXTRA1_WHITE, TR_PRIOR_CHANNEL, TR_SETTLEMENT_BANK, TR_SUB_BATCHNO_BANK, TR_SUB_BANK_NO, TR_BANK_REMARK, TR_CONTRACTTYPE, TR_SETTLEMENTBODY, TR_CHANNELBODY, TR_SETTLEMENTAMONT, TR_TSINGLEFEE, TR_TREFUNDFEE, TR_TRATE, TR_TRATERETURN, TR_TAXATION, TR_MIN_TRADE_FEE, TR_BANK_RATE_HKD, TR_SETT_RATE_HKD, TR_RETURN_AVS, TR_RETURN_CVV, TR_TYPE, TR_SETT_STATUS, TR_REFUND_LOCK, TR_MCC, TR_UPDATE_ORDER_STATUS, TR_MANUAL_LIQUIDATION_REMARK, TR_OP_ENTITY, TR_TRADE_BANK_NAME, TR_CLEAR_RATE, TR_INTERCHANGE_FEE, TR_SCHEME_FEE, TR_ACQUIRER_FEE, TR_CLEAR_TRADE_RATE, TR_CROSS_BORDER_FEE, TR_AVS_OUT_RESULT, TR_CAVV, TR_BANK_PID, TR_EXPIRATION_TIME, TR_TCHARGEBACK_FEE, TR_TCONGEAL_FEE, TR_FXAJD_FEE, TR_TOTHER_FEE) values (787160, '200818181648559171322', '1597745803378', 191713, 19171301, 'USD', 0.01, -1, 0.01, 'USD', 0, 0, NULL, NULL, NULL, 0.1, 1, 'USD', 0.01, 'OPBank', 121, 0, 0.01, 'USD', 1.0, NULL, NULL, NULL, NULL, '2020-08-18 18:16:48.139000000', NULL, '2020-08-18 18:16:48.346618000', NULL, NULL, 'http://192.168.11.98:7105/Test/jsp/testPayTwo.jsp', 'http://192.168.11.98:7105/PaymentGateway/gateway/directservice/pay', 0, NULL, 0, 0, 0, NULL, 0, 0, 0, 0, NULL, 0, 0, 0, 0, 0, 0, NULL, NULL, NULL, NULL, 0, 0, 0, 0, 0, 0, -2, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, '2fang', NULL, 1, '200818181648559171322', 0, NULL, 0, 98.0, NULL, 1, 0, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, '2020-08-18 18:16:48.346618000', 1, 0.01, 'USD', NULL, NULL, '0', NULL, 0, 0, NULL, NULL, 0.01, 0, 0, NULL, 0, 0, 0, 'OPBANK 账单地址', NULL, NULL, NULL, 1, 0, NULL, NULL, NULL, 0, NULL, NULL, 0, NULL, NULL, NULL, NULL, 302, 8, 2, NULL, NULL, NULL, NULL, NULL, NULL, 0, 7.75081, 7.75081, NULL, NULL, 1, NULL, NULL, 5722, 0, NULL, 1, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
update CCPS_TRADERECORD set TR_STATUS = 1,TR_BANKORDERNO = NULL,TR_BANKRETURNCODE = '80000',TR_BANKINFO = 'Transaction Approved1',TR_BANKDATETIME = '2020-08-18 18:16:48.360706000',TR_CHECKED = 0,TR_CHECKDATETIME = '2020-08-18 18:16:48.000000000',TR_QUERYNO = NULL,TR_AUTHORIZELD = NULL,TR_BATCHNO = NULL,TR_TERMINALNO = NULL,TR_AUTH_TYPE = 0,TR_DEBITTIME = '2020-08-18 18:16:48.360706000',TR_TRANRETURNCODE = '10000',TR_TRANINFO = 'Transaction Approved',TR_WHETHER_3D = '0',TR_ECI_VALUE = NULL,TR_NOTICESTATUS = 2,TR_PAY_SOLUTIONS = 'Transaction Approved3',TR_BANK_REMARK = ';;',TR_RETURN_AVS = NULL,TR_RETURN_CVV = NULL,TR_UPDATE_ORDER_STATUS = 0 where TR_NO = '200818181648559171322'
update CCPS_TRADERECORD set TR_RS_ID = 1000779,TR_RS_STATUS = 2,TR_UPDATE_ORDER_STATUS = 0 where TR_NO = '160627142027150120836'
代码语言:javascript
代码运行次数:0
运行
复制

在GP数据库验证结果(测试数据及DML较多,随机抽了一条验证):

代码语言:javascript
代码运行次数:0
运行
AI代码解释
复制
devin=> select TR_STATUS,TR_BANKRETURNCODE,TR_BANKDATETIME,TR_CHECKDATETIME,TR_TRANINFO,TR_BANK_REMARK from CCPS_TRADERECORD where TR_NO = '200818181648559171322';
 tr_status | tr_bankreturncode |      tr_bankdatetime       |  tr_checkdatetime   |     tr_traninfo      | tr_bank_remark 
-----------+-------------------+----------------------------+---------------------+----------------------+----------------
         1 | 80000             | 2020-08-18 18:16:48.360706 | 2020-08-18 18:16:48 | Transaction Approved | ;;
(1 row)
代码语言:javascript
代码运行次数:0
运行
复制

至此,数据同步成功。

作者

肖杰:云和恩墨Oracle技术顾问,长期服务于银行,金融,能源等行业的数据中心,8年数据库运维经验,擅长Oracle,PostgreSQL等数据库高可用设计和运维故障处理,异常恢复,升级迁移,性能优化。

墨天轮原文链接:https://www.modb.pro/db/29348(复制到浏览器中打开或者点击“阅读原文”立即查看)。

本文参与 腾讯云自媒体同步曝光计划,分享自微信公众号。
原始发表:2021-02-22,如有侵权请联系 cloudcommunity@tencent.com 删除

本文分享自 数据和云 微信公众号,前往查看

如有侵权,请联系 cloudcommunity@tencent.com 删除。

本文参与 腾讯云自媒体同步曝光计划  ,欢迎热爱写作的你一起参与!

评论
登录后参与评论
暂无评论
推荐阅读
springboot Actuator
springboot Actuator只需要加入依赖即可使用: <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> application.properties 中可以加入以下配置: # false,表示不敏感,可以随意访问,否则就是做了一些保护,不能随意访问。 endpoints.mapping
yawn
2018/03/14
6960
第11章 Spring Boot应用监控第11章 Spring Boot应用监控小结
在实际的生产系统中,我们怎样知道我们的应用运行良好呢?我们往往需要对系统实际运行的情况(各种cpu,io,disk,db,业务功能等指标)进行监控运维。这需要耗费我们不少精力来搞这些工作。在SpringBoot中,我们完全不需要面对这样的难题。
一个会写诗的程序员
2018/08/20
1.4K0
springBoot系列教程01:elasticsearch的集成及使用
 1.首先安装elasticsearch 集群环境,参考 http://www.cnblogs.com/xiaochangwei/p/8033773.html
肖哥哥
2018/08/02
1.1K0
springBoot系列教程01:elasticsearch的集成及使用
Spring Boot :四大神器之Actuator
Spring Boot有四大神器,分别是auto-configuration、starters、cli、actuator,本文主要讲actuator。actuator是spring boot提供的对应用系统的自省和监控的集成功能,可以对应用系统进行配置查看、相关功能统计等。
Freedom123
2024/03/29
1750
Spring Boot :四大神器之Actuator
SOFARPC源码解析-搭建环境
简介摘要 SOFA 是蚂蚁金服自主研发的金融级分布式中间件,包含构建金融级云原生架构所需的各个组件,包括微服务研发框架,RPC 框架,服务注册中心,分布式定时任务,限流/熔断框架,动态配置推送,分布式链路追踪,Metrics监控度量,分布式高可用消息队列,分布式事务框架,分布式数据库代理层等组件,是一套分布式架构的完整的解决方案,也是在金融场景里锤炼出来的最佳实践。 SOFARPC是蚂蚁金服开源的高可扩展性、高性能、生产级的Java RPC框架。SOFARPC致力于简化应用之间的RPC调用,为应用提供方便透明、稳定高效的点对点远程服务调用方案。为了用户和开发者方便的进行功能扩展,SOFARPC提供丰富的模型抽象和可扩展接口,包括过滤器、路由、负载均衡等等。 SOFARPC功能特性:(1)透明化、高性能的远程服务调用;(2)支持多种服务路由及负载均衡策略;(3)支持多种注册中心的集成;(4)支持多种协议;(5)支持同步、单向、回调、泛化等多种调用方式;(6)支持集群容错、服务预热、自动故障隔离;(7)强大的扩展功能,可以按需扩展各个功能组件。 SOFARPC Github:https://github.com/alipay/sofa-rpc 架构设计 SOFARPC从下到上分为两层:核心层:包含RPC 的核心组件(例如我们的各种接口、API、公共包)以及一些通用的实现(例如随机等负载均衡算法)。功能实现层:所有的功能实现层的用户都是平等的,都是基于扩展机制实现的。
爱撸猫的杰
2019/03/28
1.3K0
SOFARPC源码解析-搭建环境
Spring Boot的应用启动与关闭
Spring Boot应用可以打成jar包,其中内嵌tomcat,因此可以直接启动使用。但是在Spring Boot应用启动之前,首先需要进行打包,本文讲述的是Maven工程的打包,打包需要的前提条件(pom.xml文件中的内容)是:
Tyan
2019/05/25
1.7K0
Spring Boot Admin 2 值得了解的新变化
6.19号,spring团队发布了期待已久的 Spring Cloud Finchley.RELEASE 版本。 期间Spring Boot Admin 也发布了 2.0.1 兼容它,我在升级pig 到Finchley.RELEASE的同时 发现很多有意思的变化整理发出来
冷冷
2018/06/28
1.2K0
01-如何在 Spring Boot 应用程序中使用 Actuator 监控和管理端点,提高应用程序的生产力?
健康信息可以检查应用的运行状态,它经常被监控软件用来提醒人们生产环境是否存在问题。health端点暴露的默认信息取决于端点是如何被访问的。
JavaEdge
2023/05/24
9890
01-如何在 Spring Boot 应用程序中使用 Actuator 监控和管理端点,提高应用程序的生产力?
SpringBoot入门建站全系列(二十九)Actuator服务监控及SpringBootMonitor单机监控页面
Actuaotr是spring boot项目中非常强大的一个功能,有助于对应用程序进行监控和管理,通过restful api请求来监管、审计、收集应用的运行情况,针对微服务而言它是必不可少的一个环节。
品茗IT
2020/05/28
2.4K0
《Kotin 极简教程》第11章 使用Kotlin 集成 SpringBoot开发Web服务端第11章 使用Kotlin集成SpringBoot开发Web服务端《Kotlin极简教程》正式上架:
我们在前面第2章 “ 2.3 Web RESTFul HelloWorld ” 一节中,已经介绍了使用 Kotlin 结合 SpringBoot 开发一个RESTFul版本的 Hello World。当然,Kotlin与Spring家族的关系不止如此。在 Spring 5.0 M4 中引入了一个专门针对Kotlin的支持。
一个会写诗的程序员
2018/08/17
3.2K0
《Kotin 极简教程》第11章 使用Kotlin 集成 SpringBoot开发Web服务端第11章 使用Kotlin集成SpringBoot开发Web服务端《Kotlin极简教程》正式上架:
SpringBoot监控工具之《Actuator》
有句话怎么说呢?学的越多,不知道的就越多,现在感觉学的越来越废了,不学又不行,最近一直在钻研 SpringBoot相关的内容,准备先写一些基础的案例代码,后续研究的更加透彻,写一些有内涵的刨析文章发表出来,下面开始吧!
框架师
2021/11/22
1.7K0
SpringBoot监控工具之《Actuator》
Spring Boot Actuator监控使用详解
版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
程序新视界
2019/11/20
1.9K0
SpringBoot2.0 基础案例(16):配置Actuator组件,实现系统监控
在生产环境中,需要实时或定期监控服务的可用性。Spring Boot的actuator(健康监控)功能提供了很多监控所需的接口,可以对应用系统进行配置查看、相关功能统计等。
知了一笑
2019/07/19
5350
SpringCloud 2.x学习笔记:22、使用Sentinel实现接口限流
https://github.com/alibaba/Sentinel/releases
程裕强
2019/07/11
1.8K0
SpringCloud 2.x学习笔记:22、使用Sentinel实现接口限流
原 荐 SpringBoot 2.0 系列0
SpringBoot 2.0 系列004 --启动实战之配置文件 配置文件 配置文件加载流程 很多文档包括官方文档说SB的默认配置文件是application开头的文件,那么是为什么呢? 我们先
石奈子
2018/06/13
8900
Spring Boot 2.0 新特性(二):新增事件ApplicationStartedEvent
今天继续来聊Spring Boot 2.0的新特性。本文将具体说说2.0版本中的事件模型,尤其是新增的事件: ApplicationStartedEvent。 原文首发:http://blog.didispace.com/Spring-Boot-2-0-feature-2-ApplicationStartedEvent/ 在Spring Boot 2.0中对事件模型做了一些增强,主要就是增加了 ApplicationStartedEvent事件,所以在2.0版本中所有的事件按执行的先后顺序如下: Appli
程序猿DD
2018/03/21
2K0
聊聊springcloud的serviceRegistryEndpoint
本文主要研究一下springcloud的serviceRegistryEndpoint
code4it
2018/09/17
1.4K0
第一节,创建第一个springboot项目
版权声明:本文为博主原创文章,未经博主允许不得转载。 本系列使用工具 IDEA 1.创建项目 2.新创建完项目目录 3.开启项目 有键打开 执行 如下图,就代表启动成功了 . __
DencyCheng
2018/11/05
7650
Spring Boot Actuator详解与深入应用(二):Actuator 2.x
本文系《Spring Boot Actuator详解与深入应用》中的第二篇。在上一篇文章:Spring Boot Actuator详解与深入应用(一):Actuator 1.x主要讲了Spring Boot Actuator 1.x的应用与定制端点。Spring Boot2.0的正式版已经发布有一段时间了,目前已经到了2.1.0.RELEASE。关于Spring Boot2.x的特性,在此不详细叙述了,但是其流行的趋势是显而易见的。
aoho求索
2018/12/17
2.5K0
微服务架构之Spring Boot(七十三)
端点上的操作通过其参数接收输入。通过Web公开时,这些参数的值取自URL的查询参数和JSON请求体。通过JMX公开时,参数将映射到
用户1289394
2022/05/23
2.3K0
推荐阅读
相关推荐
springboot Actuator
更多 >
领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档