Elasticsearch 写入优化,从 3000 到 8000/s,让你的 ES 飞起来。。。(Elasticsearch write optimization, from 3000 to 8000 / s, let your es fly…)

背景

  • 基于elasticsearch-5.6.0
  • 机器配置:3个云ecs节点,16G,4核,机械硬盘

优化前,写入速度平均,一遇到压测,写入速度骤降,甚至es直接频率gc、oom等;优化后,写入速度平均,遇到压测,能在压测结束后30分钟内消化完数据,各项指标回归正常。

3000条/s
8000条/s

生产配置

这里我先把自己优化的结果贴出来,后面有参数的详解:

elasticsearch.yml中增加如下设置

indices.memory.index_buffer_size: 20%
indices.memory.min_index_buffer_size: 96mb

# Search pool
thread_pool.search.size: 5
thread_pool.search.queue_size: 100
# 这个参数慎用!强制修改cpu核数,以突破写线程数限制
# processors: 16
# Bulk pool
#thread_pool.bulk.size: 16
thread_pool.bulk.queue_size: 300
# Index pool
#thread_pool.index.size: 16
thread_pool.index.queue_size: 300

indices.fielddata.cache.size: 40%

discovery.zen.fd.ping_timeout: 120s
discovery.zen.fd.ping_retries: 6
discovery.zen.fd.ping_interval: 30s

索引优化配置:

PUT /_template/elk
{
      "order": 6,
      "template": "logstash-*",    #这里配置模板匹配的Index名称
      "settings": {
        "number_of_replicas" : 0,    #副本数为0,需要查询性能高可以设置为1
        "number_of_shards" :   6,    #分片数为6, 副本为1时可以设置成5
         "refresh_interval": "30s",
         "index.translog.durability": "async",
        "index.translog.sync_interval": "30s"

      }
}

优化参数详解

精细设置全文域: string类型字段默认会分词,不仅会额外占用资源,而且会影响创建索引的速度。所以,把不需要分词的字段设置为

not_analyzed

禁用_all字段: 对于日志和apm数据,目前没有场景会使用到

副本数量设置为0: 因为我们目前日志数据和apm数据在es只保留最近7天的量,全量日志保存在hadoop,可以根据需要通过spark读回到es – 况且副本数量是可以随时修改的,区别分片数量

使用es自动生成id: es对于自动生成的id有优化,避免了版本查找。因为其生成的id是唯一的

设置index.refresh_interval: 索引刷新间隔,默认为1s。因为不需要如此高的实时性,我们修改为30s – 扩展学习:刷新索引到底要做什么事情

设置段合并的线程数量:

curl -XPUT 'your-es-host:9200/nginx_log-2018-03-20/_settings' -d '{ 
   "index.merge.scheduler.max_thread_count" : 1
}'

段合并的计算量庞大,而且还要吃掉大量磁盘I/O。合并在后台定期操作,因为他们可能要很长时间才能完成,尤其是比较大的段

机械磁盘在并发I/O支持方面比较差,所以我们需要降低每个索引并发访问磁盘的线程数。这个设置允许个线程同时进行磁盘操作,也就是设置为1允许三个线程

max_thread_count + 2

扩展学习:什么是段(segment)?如何合并段?为什么要合并段?(what、how、why)另外,ES 系列面试题和答案全部整理好了,微信搜索Java技术栈,在后台发送:面试,可以在线阅读。

扩展学习:什么是段(segment)?如何合并段?为什么要合并段?(what、how、why)另外,ES 系列面试题和答案全部整理好了,微信搜索Java技术栈,在后台发送:面试,可以在线阅读。

1.设置异步刷盘事务日志文件:

"index.translog.durability": "async",
"index.translog.sync_interval": "30s"

对于日志场景,能够接受部分数据丢失。同时有全量可靠日志存储在hadoop,丢失了也可以从hadoop恢复回来

2.elasticsearch.yml中增加如下设置:

indices.memory.index_buffer_size: 20%
indices.memory.min_index_buffer_size: 96mb

已经索引好的文档会先存放在内存缓存中,等待被写到到段(segment)中。缓存满的时候会触发段刷盘(吃i/o和cpu的操作)。默认最小缓存大小为48m,不太够,最大为堆内存的10%。对于大量写入的场景也显得有点小。

扩展学习:数据写入流程是怎么样的(具体到如何构建索引)?

扩展学习:数据写入流程是怎么样的(具体到如何构建索引)?

1.设置index、merge、bulk、search的线程数和队列数。例如以下elasticsearch.yml设置:

# Search pool
thread_pool.search.size: 5
thread_pool.search.queue_size: 100
# 这个参数慎用!强制修改cpu核数,以突破写线程数限制
# processors: 16
# Bulk pool
thread_pool.bulk.size: 16
thread_pool.bulk.queue_size: 300
# Index pool
thread_pool.index.size: 16
thread_pool.index.queue_size: 300

2.设置filedata cache大小,例如以下elasticsearch.yml配置:

indices.fielddata.cache.size: 15%

filedata cache的使用场景是一些聚合操作(包括排序),构建filedata cache是个相对昂贵的操作。所以尽量能让他保留在内存中

然后日志场景聚合操作比较少,绝大多数也集中在半夜,所以限制了这个值的大小,默认是不受限制的,很可能占用过多的堆内存

扩展学习:什么是filedata?构建流程是怎样的?为什么要用filedata?(what、how、why)

扩展学习:什么是filedata?构建流程是怎样的?为什么要用filedata?(what、how、why)

1.设置节点之间的故障检测配置,例如以下elasticsearch.yml配置:

discovery.zen.fd.ping_timeout: 120s
discovery.zen.fd.ping_retries: 6
discovery.zen.fd.ping_interval: 30s

大数量写入的场景,会占用大量的网络带宽,很可能使节点之间的心跳超时。并且默认的心跳间隔也相对过于频繁(1s检测一次)

此项配置将大大缓解节点间的超时问题

后记

这里仅仅是记录对我们实际写入有提升的一些配置项,没有针对个别配置项做深入研究。

扩展学习后续填坑。基本都遵循(what、how、why)原则去学习。

版权声明:本文为CSDN博主「无影V随风」的原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接及本声明。原文链接:https://blog.csdn.net/wmj2004/article/details/80804411

版权声明:本文为CSDN博主「无影V随风」的原创文章,遵循CC 4.0 BY-SA版权协议,转载请附上原文出处链接及本声明。原文链接:https://blog.csdn.net/wmj2004/article/details/80804411

近期热文推荐:

1.1,000+ 道 Java面试题及答案整理(2022最新版)

2.劲爆!Java 协程要来了。。。

3.Spring Boot 2.x 教程,太全了!

4.别再写满屏的爆爆爆炸类了,试试装饰器模式,这才是优雅的方式!!

5.《Java开发手册(嵩山版)》最新发布,速速下载!

觉得不错,别忘了随手点赞+转发哦!

————————

background

  • 基于elasticsearch-5.6.0
  • Machine configuration: 3 cloud ECS nodes, 16g, 4-core, mechanical hard disk

Before optimization, the writing speed is average. In case of voltage measurement, the writing speed drops sharply, even es direct frequency GC, oom, etc; After optimization, the writing speed is average. In case of pressure measurement, the data can be digested within 30 minutes after the pressure measurement, and all indicators return to normal.

3000条/s
8000条/s

Production configuration

Here, I’ll post my optimization results first, followed by a detailed explanation of the parameters:

elasticsearch. Add the following settings to YML

indices.memory.index_buffer_size: 20%
indices.memory.min_index_buffer_size: 96mb

# Search pool
thread_pool.search.size: 5
thread_pool.search.queue_size: 100
# 这个参数慎用!强制修改cpu核数,以突破写线程数限制
# processors: 16
# Bulk pool
#thread_pool.bulk.size: 16
thread_pool.bulk.queue_size: 300
# Index pool
#thread_pool.index.size: 16
thread_pool.index.queue_size: 300

indices.fielddata.cache.size: 40%

discovery.zen.fd.ping_timeout: 120s
discovery.zen.fd.ping_retries: 6
discovery.zen.fd.ping_interval: 30s

Index optimization configuration:

PUT /_template/elk
{
      "order": 6,
      "template": "logstash-*",    #这里配置模板匹配的Index名称
      "settings": {
        "number_of_replicas" : 0,    #副本数为0,需要查询性能高可以设置为1
        "number_of_shards" :   6,    #分片数为6, 副本为1时可以设置成5
         "refresh_interval": "30s",
         "index.translog.durability": "async",
        "index.translog.sync_interval": "30s"

      }
}

Detailed explanation of optimization parameters

< strong > fine setting of full-text field: < / strong > the string type field will be segmented by default, which will not only occupy additional resources, but also affect the speed of index creation. Therefore, set the fields that do not need word segmentation to

not_analyzed

< strong > disable_ All field: < / strong > for log and APM data, no scenario will be used at present

< strong > the number of copies is set to 0: < / strong > at present, the log data and APM data are only kept in ES for the last 7 days. The full amount of logs are saved in Hadoop and can be read back to es through spark as needed – in addition, the number of copies can be modified at any time to distinguish the number of slices

< strong > use es to automatically generate ID: < / strong > es optimizes the automatically generated ID to avoid version search. Because the generated ID is unique

< strong > set index refresh_ Interval: < / strong > index refresh interval, which is 1s by default. Because we don’t need such high real-time performance, we modify it to 30s – Extended Learning: what is it to do to refresh the index

< strong > set the number of threads for segment merging: < / strong >

curl -XPUT 'your-es-host:9200/nginx_log-2018-03-20/_settings' -d '{ 
   "index.merge.scheduler.max_thread_count" : 1
}'

The calculation of segment merging is huge, and it also consumes a lot of disk I / O. Merges operate regularly in the background because they may take a long time to complete, especially in large segments

Mechanical disks have poor concurrent I / O support, so we need to reduce the number of threads accessing the disk concurrently per index. This setting allows multiple threads to perform disk operations at the same time, that is, set to 1 to allow three threads

max_thread_count + 2

Extended learning: what is segment? How do I merge segments? Why merge segments? (what, how, why) in addition, the ES series interview questions and answers are all sorted out. Wechat searches the Java technology stack and sends them in the background: the interview can be read online.

Extended learning: what is segment? How do I merge segments? Why merge segments? (what, how, why) in addition, the ES series interview questions and answers are all sorted out. Wechat searches the Java technology stack and sends them in the background: the interview can be read online.

1. Set asynchronous disk flushing transaction log file:

"index.translog.durability": "async",
"index.translog.sync_interval": "30s"

For log scenarios, partial data loss can be accepted. At the same time, a full amount of reliable logs are stored in Hadoop. If they are lost, they can also be recovered from Hadoop

2.elasticsearch. Add the following settings to YML:

indices.memory.index_buffer_size: 20%
indices.memory.min_index_buffer_size: 96mb

The indexed documents will be stored in the memory cache before being written to the segment. When the cache is full, it will trigger segment disk brushing (I / O and CPU operations). The default minimum cache size is 48m, which is not enough. The maximum cache size is 10% of heap memory. It is also a little small for scenes with a large number of writes.

Extended learning: what is the data writing process (specific to how to build an index)?

Extended learning: what is the data writing process (specific to how to build an index)?

1. Set the number of threads and queues of index, merge, bulk and search. For example, the following elasticsearch YML settings:

# Search pool
thread_pool.search.size: 5
thread_pool.search.queue_size: 100
# 这个参数慎用!强制修改cpu核数,以突破写线程数限制
# processors: 16
# Bulk pool
thread_pool.bulk.size: 16
thread_pool.bulk.queue_size: 300
# Index pool
thread_pool.index.size: 16
thread_pool.index.queue_size: 300

2. Set the size of the filedata cache, such as the following elasticsearch YML configuration:

indices.fielddata.cache.size: 15%

The usage scenario of filedata cache is some aggregation operations (including sorting), and building filedata cache is a relatively expensive operation. So try to keep it in memory

Then, there are few log scene aggregation operations, and most of them are concentrated in the middle of the night, so the size of this value is limited. By default, it is unlimited, which is likely to occupy too much heap memory

Extended learning: what is filedata? What is the build process? Why use filedata? (what、how、why)

Extended learning: what is filedata? What is the build process? Why use filedata? (what、how、why)

1. Set the fault detection configuration between nodes, such as the following elasticsearch YML configuration:

discovery.zen.fd.ping_timeout: 120s
discovery.zen.fd.ping_retries: 6
discovery.zen.fd.ping_interval: 30s

The scenario with a large number of writes will occupy a large amount of network bandwidth, which is likely to make the heartbeat between nodes timeout. And the default heartbeat interval is relatively too frequent (once every 1s)

This configuration will greatly alleviate the timeout problem between nodes

Postscript

Here are only some configuration items that have improved our actual writing. There is no in-depth study on individual configuration items.

Extended learning and subsequent pit filling. Basically follow the principles of (what, how, why) to learn.

Copyright notice: This is the original article of CSDN blogger “shadowless V with the wind”, which follows the CC 4.0 by-sa copyright agreement. For reprint, please attach the source link of the original text and this notice. Original link: https://blog.csdn.net/wmj2004/article/details/80804411

Copyright notice: This is the original article of CSDN blogger “shadowless V with the wind”, which follows the CC 4.0 by-sa copyright agreement. For reprint, please attach the source link of the original text and this notice. Original link: https://blog.csdn.net/wmj2004/article/details/80804411

< strong > recent hot article recommendations: < / strong >

1.1000 + java interview questions and answers (2022 latest version)

2. Strong! The Java collaboration is coming…

3.Spring Boot 2. X tutorial, too complete!

4. Don’t write the explosive category full of screen, try the decorator mode, this is the elegant way!!

5. Java Development Manual (Songshan version) is the latest release. Download it quickly!

Feel good, don’t forget to like + forward!