Postgresql同步数据到es需要借助中间工具连接器，连接器部署主要有两种方式，一种是基于Elastic云托管的连接器(Native connectors), 另外一种自己安装管理的连接器(self-managed connector).
托管方式连接器的使用方法文档：https://www.elastic.co/guide/en/enterprise-search/8.13/native-connectors.html
本文主要介绍如何自己安装配置连接器：
前提条件：
- Linux 或者 macOS系统
- Python 3.10或者更高版本
- 系统安装了Docker与Docker compose
我们在启动连接器的时候，要根据提示来设置连接器配置，所以必做还要安装 `build-essential` , 否则会报错
sudo apt install build-essential
下载连接器仓库源码：
git clone https://github.com/elastic/connectors.git && cd connectors
cp config.yml.example config.yml # 要先手动复制一份配置文件并命名为config.yml，否则后面运行脚本会报找不到这个文件
执行安装脚本
k3s@k3s-test-master:~/connectors$ ELASTIC_PASSWORD="my_new_password" CONNECTORS_VERSION=8.11.2.0 ./scripts/stack/run-stack.sh --no-snapshot
~/connectors/scripts/stack ~/connectors
Using compose file at: /home/k3s/connectors/scripts/stack/docker/docker-compose.yml
set PROJECT_ROOT to /home/k3s/connectors
ELASTICSEARCH_VERSION=8.11.2
KIBANA_VERSION=8.11.2
CONNECTORS_VERSION=8.11.2.0
Starting Elasticsearch...
Creating network "docker_connectors_stack_net" with the default driver
Creating docker_elasticsearch_1 ... done
Connecting to Elasticsearch on http://localhost:9200
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Waiting for Elasticsearch to be running...
Starting Kibana...
docker_elasticsearch_1 is up-to-date
Creating docker_kibana_1 ... done
Connecting to Kibana on http://localhost:5601
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Waiting for Kibana to be running...
Updating Kibana password in Elasticsearch running on http://localhost:9200
{ }
Do you want to run the configurator? (y/n) y
ELASTICSEARCH_VERSION=8.11.2
KIBANA_VERSION=8.11.2
CONNECTORS_VERSION=8.11.2.0
config.yml already exists in /home/k3s/connectors/scripts/stack/connectors-config. Not overwriting.
ELASTICSEARCH_VERSION=8.11.2
KIBANA_VERSION=8.11.2
CONNECTORS_VERSION=8.11.2.0
创建完成，验证：
k3s@k3s-test-master:~/connectors$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4afc1d4bb5fd docker.elastic.co/enterprise-search/elastic-connectors:8.11.2.0 "/app/bin/elastic-in…" 7 minutes ago Up 7 minutes docker_elastic-connectors_1 a0779b7bf5a6 docker.elastic.co/kibana/kibana:8.11.2 "/bin/tini -- /usr/l…" 19 minutes ago Up 19 minutes 0.0.0.0:5601->5601/tcp, :::5601->5601/tcp docker_kibana_1 f21bf94c223b docker.elastic.co/elasticsearch/elasticsearch:8.11.2 "/bin/tini -- /usr/l…" 20 minutes ago Up 20 minutes 0.0.0.0:9200->9200/tcp, :::9200->9200/tcp, 9300/tcp docker_elasticsearch_1
同步完成后页面会有提示，或者从连接器的日志也可以看出来
k3s@k3s-test-master:~/connectors$ docker logs 4afc [FMWK][14:40:33][INFO] Running connector service version 8.11.2.0 [FMWK][14:40:33][INFO] Loading config from /config/config.yml [FMWK][14:40:33][INFO] Running preflight checks [FMWK][14:40:33][INFO] Waiting for NodeConfig(scheme='http', host='localhost', port=9200, path_prefix='', headers={}, connections_per_node=10, request_timeout=10.0, http_compress=False, verify_certs=True, ca_certs=None, client_cert=None, client_key=None, ssl_assert_hostname=None, ssl_assert_fingerprint=None, ssl_version=None, ssl_context=None, ssl_show_warn=True, _extras={}) (so far: 0 secs) [FMWK][14:40:33][INFO] Extraction service is not configured, skipping its preflight check. [FMWK][14:40:33][INFO] Job Scheduling Service started, listening to events from http://localhost:9200 [FMWK][14:40:33][INFO] Job Execution Service started, listening to events from http://localhost:9200 [FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Executing full sync [FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Filtering validation started [FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Collecting local document ids [FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Iterating on remote documents [FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Sync progress -- created: 0 | updated: 0 | deleted: 0 [FMWK][13:19:08][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Sync progress -- created: 0 | updated: 100 | deleted: 0 [FMWK][13:19:09][INFO] [Connector id: z0F3jY4BelYPyai-qYpX, index name: search-bc0, Sync job id: 40GDj44BelYPyai-MYrM] Sync ended with status completed -- created: 0 | updated: 131 | deleted: 0 (took 1 seconds)
Select a connector 页面选择你的数据源类型，点击 "Continue"
输入索引名字，然后“Continue"
点击”Generate API key" 创建一个api
把生成的新的连接配置填入连接器配置文件：
k3s@k3s-test-master:~/connectors$ cat scripts/stack/connectors-config/config.yml connectors: - api_key: eUVHa2lvNEJlbFlQeWFpLXhJcmY6S1BDTHBHZElTQkNzMVpoUzFveFJjUQ==connector_id: ykGkio4BelYPyai-xYrGservice_type: postgresql - connector_id: "5EGLj44BelYPyai-lIpr"service_type: "postgresql"api_key: "NVVHTGo0NEJlbFlQeWFpLTdZclM6VEpOZEwwTWhSTm1pUDJLdGExXzdvdw==" elasticsearch.host: http://localhost:9200 elasticsearch.password: my_new_password elasticsearch.username: elastic
k3s@k3s-test-master:~/connectors$ docker restart docker_elastic-connectors_1
docker_elastic-connectors_1
重启完成后就可以回到刚才新建的索引的 “Configuration"第4步，填入你需要连接的数据库相关信息，保存即可
