Elasticsearch on Colben Notes

Elasticsearch 笔记

Wed, 30 Oct 2019 11:49:53 +0800

索引

查看某节点的全部索引

`1`	`curl http://127.0.0.1:9200/_cat/indices?v`

新建 index

`1`	`curl -X PUT http://127.0.0.1:9200/index_name`

删除 index

`1`	`curl -X DELETE http://127.0.0.1:9200/index_name`

记录

新增记录(指定记录id)

curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/doc_id -d '
{
    "aa": "11",
    "bb": "22"
}'

新增记录(不指定记录id)

curl -X POST -H "Content-Type: application/json" http://127.0.0.1:9200/index_name -d '
{
    "aa": "11",
    "bb": "22"
}'

查看记录

`1`	`curl http://127.0.0.1:9200/index_name/doc_id?pretty=true`

删除记录

`1`	`curl -X DELETE http://127.0.0.1:9200/index_name/doc_id`

更新记录

curl -X PUT -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/doc_id -d '
{
    "aa": "33",
    "bb": "44"
}'

查询

查询所有记录

`1`	`curl http://127.0.0.1:9200/index_name/_search`

查询匹配

curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
{
    "query": {"match": {"key_name": "value_pattern"}}
}'

从位置2(默认0)开始查询8(默认10)条记录

curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
{
    "query": {"match": {"key_name": "value_pattern"}},
    "from": 2,
    "size": 8
}'

逻辑 or 查询

curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
{
    "query": {"match": {"key_name": "value_pattern_1 value_pattern_2"}}
}'

逻辑 and 查询

curl -H "Content-Type: application/json" http://127.0.0.1:9200/index_name/_search -d '
{
    "query": {
        "bool": {
            "must": [
                {"match": {"key_name": "value_pattern_1"}},
                {"match": {"key_name": "value_pattern_2"}}
            ]
        }
    }
}'

区间查询

set -euo pipefail
export START_TIME="$(date +%s -d $1)"
export END_TIME="$(date +%s -d $2)"
curl -s -H "Content-Type: application/json" -o result.txt \
    http://127.0.0.1:9200/wangmei_raw/_search?pretty -d @- <<EOF
{
  "_source": [
    "spider_name",
    "spider_time",
    "media_name",
    "publish_time"
  ],
  "query": {
    "bool": {
      "filter": {
        "range": {
          "spider_time": {
            "gt": $START_TIME,
            "lte": $END_TIME
          }
        }
      }
    }
  },
  "size": 10000
}
EOF

Kibana

lucene 正则查询

1
2

#查询包含10000-99999毫秒的 message 字段
{ "regexp": { "message": "[0-9]{5}ms" } }

Dev tool 模拟 pipeline

POST _ingest/pipeline/_simulate
{
  "pipeline" : {
    "description": "",
    "processors": [
      {
        "grok": {
          "field": "message",
          "patterns": [
            "",
            ""
          ],
          "ignore_missing": true,
          "ignore_failure": true
        }
      }
    ]
  },
  "docs" : [
    {
      "_source": {
        "message": ""
      }
    },
    {
      "_source": {
        "message": ""
      }
    }
  ]
}

pipeline

查看已有的 pipeline

1
2
3

curl http://127.0.0.1:9200/_ingest/pipeline?pretty=true
# 指定 nginx-access
curl http://127.0.0.1:9200/_ingest/pipeline/nginx-access?pretty=true

Mysql 慢查询日志
Secure 登陆日志
Nginx access 日志

filebeat

CentOS7 安装 elasticsearch 集群

Wed, 30 Oct 2019 01:19:09 +0800

环境

主机名	IP	操作系统	ES 版本
es227	192.168.1.227	CentOS7.5	6.5.4
es228	192.168.1.228	CentOS7.5	6.5.4
es229	192.168.1.229	CentOS7.5	6.5.4

下载 elasticsearch-6.5.4.tar.gz

各节点初始配置

关闭 selinux、防火墙
部署 java 运行环境
创建 elastic 用户
1

useradd -m elastic

创建数据目录

1
2
3

cd /var/lib
mkdir elasticsearch
chown elastic.elastic elasticsearch

创建日志目录

1
2
3

cd /var/log
mkdir -p elasticsearch
chown elastic.elastic elasticsearch

增加 sysctl.conf 配置，执行 sysctl -p 生效
1

vm.max_map_count = 262144

增加 /etc/security/limits.conf 配置

elastic soft nofile 65536
elastic hard nofile 65536
elastic soft memlock unlimited
elastic hard memlock unlimited

部署 ELASTICSEARCH

登陆 es227，下载 elasticsearch，解压至 /opt/ 下

修改 elasticsearch 目录的权限

`1`	`chown -R elastic.elastic /opt/elasticsearch`

修改 jvm 参数文件 /opt/elasticsearch/config/jvm.options

修改 /opt/elsaticsearch/config/elasticsearch.yml

cluster.name: TEST_ES_CLUSTER
node.name: es227
network.host: 192.168.1.227
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
discovery.zen.ping.unicast.hosts: ["192.168.1.227:9300", "192.168.1.228:9300", "192.168.1.229:9300"]
discovery.zen.minimum_master_nodes: 2
gateway.recover_after_nodes: 2

打包 elasticsearch 目录，复制到 es228 和 es229 上，并修改 elasticsearch.yml

# es228
node.name: es228
network.host: 192.168.1.228
# es229
node.name: es229
network.host: 192.168.1.229

启动集群(两种启动方式)

直接启动二进制
- 在每个节点上启动 elasticsearch 服务
  1
  
  su - elastic -c '/opt/elasticsearch/bin/elasticsearch -d'

systemd 启动

创建文件 /usr/lib/systemd/system/elasticsearch.service，内容如下

[Unit]
Description=ElasticSearch
Requires=network.service
After=network.service
[Service]
User=elastic
Group=elastic
LimitNOFILE=65536
LimitMEMLOCK=infinity
Environment=JAVA_HOME=/opt/jre
ExecStart=/opt/elasticsearch/bin/elasticsearch
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target

启动 elasticsearch 服务

1
2

systemctl daemon-reload
systemctl start elasticsearch

查看集群状态

查看集群节点状态

# 查看节点状态
curl http://192.168.1.228:9200/_cat/nodes?pretty
# 查看集群状态
curl http://192.168.1.228:9200/_cluster/state?pretty

安装分词插件

登陆 es227，下载插件 elasticsearch-analysis-ik-6.5.4.zip 至根目录下
复制该文件至 es228 和 es229 的根目录下

每个节点上安装

1
2

su - elastic
/opt/elasticsearch/bin/elasticsearch-plugin install file:///elasticsearch-analysis-ik-6.5.4.zip

配置远程扩展字典时，出现 java.net.SocketPermission 拒绝连接，此时需配置 jre 策略

1
2

# vim /opt/jre/lib/security/java.policy，在最后一个 "}" 前追加下面一行
permission java.net.SocketPermission "*:*","accept,connect,resolve";