Merge branch 'release-1.0.1' into 'main'

.env

 ELK_VERSION=7.12.0
+ES_PORT1=19200
+ES_PORT2=19300
+LOGSTASH_PORT1=15044
+LOGSTASH_PORT2=15000
+LOGSTASH_PORT3=15000
+LOGSTASH_PORT4=19600
+KIBANA_PORT=15601
\ No newline at end of file

.idea/.gitignore

+# Default ignored files
+/shelf/
+/workspace.xml
+# Datasource local storage ignored files
+/dataSources/
+/dataSources.local.xml
+# Editor-based HTTP Client requests
+/httpRequests/

.idea/docker-elk.iml

+<?xml version="1.0" encoding="UTF-8"?>
+<module type="WEB_MODULE" version="4">
+  <component name="Go" enabled="true" />
+  <component name="NewModuleRootManager">
+    <content url="file://$MODULE_DIR$" />
+    <orderEntry type="inheritedJdk" />
+    <orderEntry type="sourceFolder" forTests="false" />
+  </component>
+</module>
\ No newline at end of file

.idea/modules.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="ProjectModuleManager">
+    <modules>
+      <module fileurl="file://$PROJECT_DIR$/.idea/docker-elk.iml" filepath="$PROJECT_DIR$/.idea/docker-elk.iml" />
+    </modules>
+  </component>
+</project>
\ No newline at end of file

.idea/vcs.xml

+<?xml version="1.0" encoding="UTF-8"?>
+<project version="4">
+  <component name="VcsDirectoryMappings">
+    <mapping directory="$PROJECT_DIR$" vcs="Git" />
+  </component>
+</project>
\ No newline at end of file

ELK部署文档.md

-# Elastic stack (ELK) 部署说明
-# Elastic stack (ELK) 部署说明
-
-本项目基于github开源项目：https://github.com/deviantony/docker-elk
-
-###测试环境 
-http://124.71.144.118:15601/app/discover
-
-### 简介
-Elasticsearch 用于数据存储
-
-Logstash 用于数据处理
-
-Kibana 用于数据的展示
-
-Filebeat 负责数据采集
-
-###安装
-```console
-$ git clone https://gitlab.33.cn/dkl/docker-elk.git
-```
-
-###配置
-检查docker-compose.yml 端口是否有冲突, java虚拟机内存(ES_JAVA_OPTS)是否足够
-
-.env 中可配置ELK的版本，启动时会自动拉取对应版本的镜像
-
-###启动
-日志分开输出，kinaba日志会比较多，合在一起不利于分析。
-```console
-$ docker-compose up elasticsearch >es.log 2>&1 &
-$ docker-compose up kinaba >kinaba.log 2>&1 &
-$ docker-compose up logstash >logstash.log 2>&1 &
-```
-
-### Filebeat 部署
-拷贝filebeat.tgz到日志文件所在服务器，并解压
-
-```console
-$ tar -zxvf filebeat.tgz
-```
-修改配置文件 filebeat.yml
-```yml
-filebeat.inputs:
-  - type: log
-    paths:
-      # **可匹配多级目录
-      - '/data/backend-micro/**/*.log'
-    exclude_files: ['stat.log$']
-
-# 注意修改为logstash安装的地址。
-output.logstash:
-  hosts: ["127.0.0.1:15044"]
-```
-启动。
-```console
-$ ./filebeat &
-```
-日志查看
-```console
-$ tail -100f logs/filebeat
-```
-配置开机自启,并启动
-```console
-$ sh install.sh
-$ sh start.sh
-```
+++ /dev/null
\ No newline at end of file

README.md

-# Elastic stack (ELK) on Docker
 
-[![Elastic Stack version](https://img.shields.io/badge/Elastic%20Stack-7.12.0-00bfb3?style=flat&logo=elastic-stack)](https://www.elastic.co/blog/category/releases)
-[![Build Status](https://github.com/deviantony/docker-elk/workflows/CI/badge.svg?branch=main)](https://github.com/deviantony/docker-elk/actions?query=workflow%3ACI+branch%3Amain)
-[![Join the chat at https://gitter.im/deviantony/docker-elk](https://badges.gitter.im/Join%20Chat.svg)](https://gitter.im/deviantony/docker-elk?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+# Elastic stack (ELK) 部署说明
 
-Run the latest version of the [Elastic stack][elk-stack] with Docker and Docker Compose.
+本项目基于github开源项目：https://github.com/deviantony/docker-elk
 
-It gives you the ability to analyze any data set by using the searching/aggregation capabilities of Elasticsearch and
-the visualization power of Kibana.
+项目地址：https://gitlab.33.cn/dkl/docker-elk
 
-*:information_source: The Docker images backing this stack include [X-Pack][xpack] with [paid features][paid-features]
-enabled by default (see [How to disable paid features](#how-to-disable-paid-features) to disable them). **The [trial
-license][trial-license] is valid for 30 days**. After this license expires, you can continue using the free features
-seamlessly, without losing any data.*
+###测试环境
+http://124.71.144.118:15601/app/discover
 
-Based on the official Docker images from Elastic:
+### 简介
+Elasticsearch 用于数据存储
 
-* [Elasticsearch](https://github.com/elastic/elasticsearch/tree/master/distribution/docker)
-* [Logstash](https://github.com/elastic/logstash/tree/master/docker)
-* [Kibana](https://github.com/elastic/kibana/tree/master/src/dev/build/tasks/os_packages/docker_generator)
+Logstash 用于数据处理
 
-Other available stack variants:
+Kibana 用于数据的展示
 
-* [`tls`](https://github.com/deviantony/docker-elk/tree/tls): TLS encryption enabled in Elasticsearch.
-* [`searchguard`](https://github.com/deviantony/docker-elk/tree/searchguard): Search Guard support
+Filebeat 负责数据采集
 
----
+###安装
 
-## Philosophy
+    wget https://gitlab.33.cn/dkl/docker-elk/-/archive/1.0.0/docker-elk-1.0.0.tar.gz
+    tar -zxvf docker-elk-1.0.0.tar.gz
 
-We aim at providing the simplest possible entry into the Elastic stack for anybody who feels like experimenting with
-this powerful combo of technologies. This project's default configuration is purposely minimal and unopinionated. It
-does not rely on any external dependency or custom automation to get things up and running.
+###启动并初始化
 
-Instead, we believe in good documentation so that you can use this repository as a template, tweak it, and make it _your
-own_. [sherifabdlnaby/elastdocker][elastdocker] is one example among others of project that builds upon this idea.
+    bash start-and-init.sh
 
----
+###日志查看
 
-## Contents
+    docker-compose logs --tail=100 elasticsearch
+    docker-compose logs --tail=100 kibana
+    docker-compose logs --tail=100 logstash
 
-1. [Requirements](#requirements)
-   * [Host setup](#host-setup)
-   * [SELinux](#selinux)
-   * [Docker for Desktop](#docker-for-desktop)
-     * [Windows](#windows)
-     * [macOS](#macos)
-1. [Usage](#usage)
-   * [Version selection](#version-selection)
-   * [Bringing up the stack](#bringing-up-the-stack)
-   * [Cleanup](#cleanup)
-   * [Initial setup](#initial-setup)
-     * [Setting up user authentication](#setting-up-user-authentication)
-     * [Injecting data](#injecting-data)
-     * [Default Kibana index pattern creation](#default-kibana-index-pattern-creation)
-1. [Configuration](#configuration)
-   * [How to configure Elasticsearch](#how-to-configure-elasticsearch)
-   * [How to configure Kibana](#how-to-configure-kibana)
-   * [How to configure Logstash](#how-to-configure-logstash)
-   * [How to disable paid features](#how-to-disable-paid-features)
-   * [How to scale out the Elasticsearch cluster](#how-to-scale-out-the-elasticsearch-cluster)
-   * [How to reset a password programmatically](#how-to-reset-a-password-programmatically)
-1. [Extensibility](#extensibility)
-   * [How to add plugins](#how-to-add-plugins)
-   * [How to enable the provided extensions](#how-to-enable-the-provided-extensions)
-1. [JVM tuning](#jvm-tuning)
-   * [How to specify the amount of memory used by a service](#how-to-specify-the-amount-of-memory-used-by-a-service)
-   * [How to enable a remote JMX connection to a service](#how-to-enable-a-remote-jmx-connection-to-a-service)
-1. [Going further](#going-further)
-   * [Plugins and integrations](#plugins-and-integrations)
-   * [Swarm mode](#swarm-mode)
+### Filebeat 部署
+项目地址： https://gitlab.33.cn/dkl/filebeat
+### 1. quick install
 
-## Requirements
+Download quick install script
 
-### Host setup
+    wget https://gitlab.33.cn/dkl/filebeat/raw/master/filebeat-quick-install.sh
 
-* [Docker Engine](https://docs.docker.com/install/) version **17.05** or newer
-* [Docker Compose](https://docs.docker.com/compose/install/) version **1.20.0** or newer
-* 1.5 GB of RAM
+Config logstash host and scan dir
 
-*:information_source: Especially on Linux, make sure your user has the [required permissions][linux-postinstall] to
-interact with the Docker daemon.*
+    vim filebeat-quick-install.sh
 
-By default, the stack exposes the following ports:
+Run this script
 
-* 5044: Logstash Beats input
-* 5000: Logstash TCP input
+    bash filebeat-quick-install.sh
\ No newline at end of file
-* 9600: Logstash monitoring API
-* 9200: Elasticsearch HTTP
-* 9300: Elasticsearch TCP transport
-* 5601: Kibana
-
-**:warning: Elasticsearch's [bootstrap checks][booststap-checks] were purposely disabled to facilitate the setup of the
-Elastic stack in development environments. For production setups, we recommend users to set up their host according to
-the instructions from the Elasticsearch documentation: [Important System Configuration][es-sys-config].**
-
-### SELinux
-
-On distributions which have SELinux enabled out-of-the-box you will need to either re-context the files or set SELinux
-into Permissive mode in order for docker-elk to start properly. For example on Redhat and CentOS, the following will
-apply the proper context:
-
-```console
-$ chcon -R system_u:object_r:admin_home_t:s0 docker-elk/
-```
-
-### Docker for Desktop
-
-#### Windows
-
-Ensure the [Shared Drives][win-shareddrives] feature is enabled for the `C:` drive.
-
-#### macOS
-
-The default Docker for Mac configuration allows mounting files from `/Users/`, `/Volumes/`, `/private/`, and `/tmp`
-exclusively. Make sure the repository is cloned in one of those locations or follow the instructions from the
-[documentation][mac-mounts] to add more locations.
-
-## Usage
-
-### Version selection
-
-This repository tries to stay aligned with the latest version of the Elastic stack. The `main` branch tracks the current
-major version (7.x).
-
-To use a different version of the core Elastic components, simply change the version number inside the `.env` file. If
-you are upgrading an existing stack, please carefully read the note in the next section.
-
-**:warning: Always pay attention to the [official upgrade instructions][upgrade] for each individual component before
-performing a stack upgrade.**
-
-Older major versions are also supported on separate branches:
-
-* [`release-6.x`](https://github.com/deviantony/docker-elk/tree/release-6.x): 6.x series
-* [`release-5.x`](https://github.com/deviantony/docker-elk/tree/release-5.x): 5.x series (End-Of-Life)
-
-### Bringing up the stack
-
-Clone this repository onto the Docker host that will run the stack, then start services locally using Docker Compose:
-
-```console
-$ docker-compose up
-```
-
-You can also run all services in the background (detached mode) by adding the `-d` flag to the above command.
-
-**:warning: You must rebuild the stack images with `docker-compose build` whenever you switch branch or update the
-version of an already existing stack.**
-
-If you are starting the stack for the very first time, please read the section below attentively.
-
-### Cleanup
-
-Elasticsearch data is persisted inside a volume by default.
-
-In order to entirely shutdown the stack and remove all persisted data, use the following Docker Compose command:
-
-```console
-$ docker-compose down -v
-```
-
-## Initial setup
-
-### Setting up user authentication
-
-*:information_source: Refer to [How to disable paid features](#how-to-disable-paid-features) to disable authentication.*
-
-The stack is pre-configured with the following **privileged** bootstrap user:
-
-* user: *elastic*
-* password: *changeme*
-
-Although all stack components work out-of-the-box with this user, we strongly recommend using the unprivileged [built-in
-users][builtin-users] instead for increased security.
-
-1. Initialize passwords for built-in users
-
-    ```console
-    $ docker-compose exec -T elasticsearch bin/elasticsearch-setup-passwords auto --batch
-    ```
-
-    Passwords for all 6 built-in users will be randomly generated. Take note of them.
-
-1. Unset the bootstrap password (_optional_)
-
-    Remove the `ELASTIC_PASSWORD` environment variable from the `elasticsearch` service inside the Compose file
-    (`docker-compose.yml`). It is only used to initialize the keystore during the initial startup of Elasticsearch.
-
-1. Replace usernames and passwords in configuration files
-
-    Use the `kibana_system` user (`kibana` for releases <7.8.0) inside the Kibana configuration file
-    (`kibana/config/kibana.yml`) and the `logstash_system` user inside the Logstash configuration file
-    (`logstash/config/logstash.yml`) in place of the existing `elastic` user.
-
-    Replace the password for the `elastic` user inside the Logstash pipeline file (`logstash/pipeline/logstash.conf`).
-
-    *:information_source: Do not use the `logstash_system` user inside the Logstash **pipeline** file, it does not have
-    sufficient permissions to create indices. Follow the instructions at [Configuring Security in Logstash][ls-security]
-    to create a user with suitable roles.*
-
-    See also the [Configuration](#configuration) section below.
-
-1. Restart Kibana and Logstash to apply changes
-
-    ```console
-    $ docker-compose restart kibana logstash
-    ```
-
-    *:information_source: Learn more about the security of the Elastic stack at [Tutorial: Getting started with
-    security][sec-tutorial].*
-
-### Injecting data
-
-Give Kibana about a minute to initialize, then access the Kibana web UI by opening <http://localhost:5601> in a web
-browser and use the following credentials to log in:
-
-* user: *elastic*
-* password: *\<your generated elastic password>*
-
-Now that the stack is running, you can go ahead and inject some log entries. The shipped Logstash configuration allows
-you to send content via TCP:
-
-```console
-# Using BSD netcat (Debian, Ubuntu, MacOS system, ...)
-$ cat /path/to/logfile.log | nc -q0 localhost 5000
-```
-
-```console
-# Using GNU netcat (CentOS, Fedora, MacOS Homebrew, ...)
-$ cat /path/to/logfile.log | nc -c localhost 5000
-```
-
-You can also load the sample data provided by your Kibana installation.
-
-### Default Kibana index pattern creation
-
-When Kibana launches for the first time, it is not configured with any index pattern.
-
-#### Via the Kibana web UI
-
-*:information_source: You need to inject data into Logstash before being able to configure a Logstash index pattern via
-the Kibana web UI.*
-
-Navigate to the _Discover_ view of Kibana from the left sidebar. You will be prompted to create an index pattern. Enter
-`logstash-*` to match Logstash indices then, on the next page, select `@timestamp` as the time filter field. Finally,
-click _Create index pattern_ and return to the _Discover_ view to inspect your log entries.
-
-Refer to [Connect Kibana with Elasticsearch][connect-kibana] and [Creating an index pattern][index-pattern] for detailed
-instructions about the index pattern configuration.
-
-#### On the command line
-
-Create an index pattern via the Kibana API:
-
-```console
-$ curl -XPOST -D- 'http://localhost:5601/api/saved_objects/index-pattern' \
-    -H 'Content-Type: application/json' \
-    -H 'kbn-version: 7.12.0' \
-    -u elastic:<your generated elastic password> \
-    -d '{"attributes":{"title":"logstash-*","timeFieldName":"@timestamp"}}'
-```
-
-The created pattern will automatically be marked as the default index pattern as soon as the Kibana UI is opened for the
-first time.
-
-## Configuration
-
-*:information_source: Configuration is not dynamically reloaded, you will need to restart individual components after
-any configuration change.*
-
-### How to configure Elasticsearch
-
-The Elasticsearch configuration is stored in [`elasticsearch/config/elasticsearch.yml`][config-es].
-
-You can also specify the options you want to override by setting environment variables inside the Compose file:
-
-```yml
-elasticsearch:
-
-  environment:
-    network.host: _non_loopback_
-    cluster.name: my-cluster
-```
-
-Please refer to the following documentation page for more details about how to configure Elasticsearch inside Docker
-containers: [Install Elasticsearch with Docker][es-docker].
-
-### How to configure Kibana
-
-The Kibana default configuration is stored in [`kibana/config/kibana.yml`][config-kbn].
-
-It is also possible to map the entire `config` directory instead of a single file.
-
-Please refer to the following documentation page for more details about how to configure Kibana inside Docker
-containers: [Install Kibana with Docker][kbn-docker].
-
-### How to configure Logstash
-
-The Logstash configuration is stored in [`logstash/config/logstash.yml`][config-ls].
-
-It is also possible to map the entire `config` directory instead of a single file, however you must be aware that
-Logstash will be expecting a [`log4j2.properties`][log4j-props] file for its own logging.
-
-Please refer to the following documentation page for more details about how to configure Logstash inside Docker
-containers: [Configuring Logstash for Docker][ls-docker].
-
-### How to disable paid features
-
-Switch the value of Elasticsearch's `xpack.license.self_generated.type` option from `trial` to `basic` (see [License
-settings][trial-license]).
-
-### How to scale out the Elasticsearch cluster
-
-Follow the instructions from the Wiki: [Scaling out Elasticsearch](https://github.com/deviantony/docker-elk/wiki/Elasticsearch-cluster)
-
-### How to reset a password programmatically
-
-If for any reason your are unable to use Kibana to change the password of your users (including [built-in
-users][builtin-users]), you can use the Elasticsearch API instead and achieve the same result.
-
-In the example below, we reset the password of the `elastic` user (notice "/user/elastic" in the URL):
-
-```console
-$ curl -XPOST -D- 'http://localhost:9200/_security/user/elastic/_password' \
-    -H 'Content-Type: application/json' \
-    -u elastic:<your current elastic password> \
-    -d '{"password" : "<your new password>"}'
-```
-
-## Extensibility
-
-### How to add plugins
-
-To add plugins to any ELK component you have to:
-
-1. Add a `RUN` statement to the corresponding `Dockerfile` (eg. `RUN logstash-plugin install logstash-filter-json`)
-1. Add the associated plugin code configuration to the service configuration (eg. Logstash input/output)
-1. Rebuild the images using the `docker-compose build` command
-
-### How to enable the provided extensions
-
-A few extensions are available inside the [`extensions`](extensions) directory. These extensions provide features which
-are not part of the standard Elastic stack, but can be used to enrich it with extra integrations.
-
-The documentation for these extensions is provided inside each individual subdirectory, on a per-extension basis. Some
-of them require manual changes to the default ELK configuration.
-
-## JVM tuning
-
-### How to specify the amount of memory used by a service
-
-By default, both Elasticsearch and Logstash start with [1/4 of the total host
-memory](https://docs.oracle.com/javase/8/docs/technotes/guides/vm/gctuning/parallel.html#default_heap_size) allocated to
-the JVM Heap Size.
-
-The startup scripts for Elasticsearch and Logstash can append extra JVM options from the value of an environment
-variable, allowing the user to adjust the amount of memory that can be used by each component:
-
-| Service       | Environment variable |
-|---------------|----------------------|
-| Elasticsearch | ES_JAVA_OPTS         |
-| Logstash      | LS_JAVA_OPTS         |
-
-To accomodate environments where memory is scarce (Docker for Mac has only 2 GB available by default), the Heap Size
-allocation is capped by default to 256MB per service in the `docker-compose.yml` file. If you want to override the
-default JVM configuration, edit the matching environment variable(s) in the `docker-compose.yml` file.
-
-For example, to increase the maximum JVM Heap Size for Logstash:
-
-```yml
-logstash:
-
-  environment:
-    LS_JAVA_OPTS: -Xmx1g -Xms1g
-```
-
-### How to enable a remote JMX connection to a service
-
-As for the Java Heap memory (see above), you can specify JVM options to enable JMX and map the JMX port on the Docker
-host.
-
-Update the `{ES,LS}_JAVA_OPTS` environment variable with the following content (I've mapped the JMX service on the port
-18080, you can change that). Do not forget to update the `-Djava.rmi.server.hostname` option with the IP address of your
-Docker host (replace **DOCKER_HOST_IP**):
-
-```yml
-logstash:
-
-  environment:
-    LS_JAVA_OPTS: -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=18080 -Dcom.sun.management.jmxremote.rmi.port=18080 -Djava.rmi.server.hostname=DOCKER_HOST_IP -Dcom.sun.management.jmxremote.local.only=false
-```
-
-## Going further
-
-### Plugins and integrations
-
-See the following Wiki pages:
-
-* [External applications](https://github.com/deviantony/docker-elk/wiki/External-applications)
-* [Popular integrations](https://github.com/deviantony/docker-elk/wiki/Popular-integrations)
-
-### Swarm mode
-
-Experimental support for Docker [Swarm mode][swarm-mode] is provided in the form of a `docker-stack.yml` file, which can
-be deployed in an existing Swarm cluster using the following command:
-
-```console
-$ docker stack deploy -c docker-stack.yml elk
-```
-
-If all components get deployed without any error, the following command will show 3 running services:
-
-```console
-$ docker stack services elk
-```
-
-*:information_source: To scale Elasticsearch in Swarm mode, configure seed hosts with the DNS name `tasks.elasticsearch`
-instead of `elasticsearch`.*
-
-[elk-stack]: https://www.elastic.co/what-is/elk-stack
-[xpack]: https://www.elastic.co/what-is/open-x-pack
-[paid-features]: https://www.elastic.co/subscriptions
-[trial-license]: https://www.elastic.co/guide/en/elasticsearch/reference/current/license-settings.html
-
-[elastdocker]: https://github.com/sherifabdlnaby/elastdocker
-
-[linux-postinstall]: https://docs.docker.com/install/linux/linux-postinstall/
-
-[booststap-checks]: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
-[es-sys-config]: https://www.elastic.co/guide/en/elasticsearch/reference/current/system-config.html
-
-[win-shareddrives]: https://docs.docker.com/docker-for-windows/#shared-drives
-[mac-mounts]: https://docs.docker.com/docker-for-mac/osxfs/
-
-[builtin-users]: https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html
-[ls-security]: https://www.elastic.co/guide/en/logstash/current/ls-security.html
-[sec-tutorial]: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-getting-started.html
-
-[connect-kibana]: https://www.elastic.co/guide/en/kibana/current/connect-to-elasticsearch.html
-[index-pattern]: https://www.elastic.co/guide/en/kibana/current/index-patterns.html
-
-[config-es]: ./elasticsearch/config/elasticsearch.yml
-[config-kbn]: ./kibana/config/kibana.yml
-[config-ls]: ./logstash/config/logstash.yml
-
-[es-docker]: https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
-[kbn-docker]: https://www.elastic.co/guide/en/kibana/current/docker.html
-[ls-docker]: https://www.elastic.co/guide/en/logstash/current/docker-config.html
-
-[log4j-props]: https://github.com/elastic/logstash/tree/7.6/docker/data/logstash/config
-[esuser]: https://github.com/elastic/elasticsearch/blob/7.6/distribution/docker/src/docker/Dockerfile#L23-L24
-
-[upgrade]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-upgrade.html
-
-[swarm-mode]: https://docs.docker.com/engine/swarm/

docker-compose.yml

@@ -15,14 +15,15 @@ services:
         source: elasticsearch
         target: /usr/share/elasticsearch/data
     ports:
-      - "19200:9200"
-      - "19300:9300"
+      - "$ES_PORT1:9200"
+      - "$ES_PORT2:9300"
     environment:
       ES_JAVA_OPTS: "-Xmx512m -Xms512m"
       ELASTIC_PASSWORD: changeme
       # Use single node discovery in order to disable production mode and avoid bootstrap checks.
       # see: https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
       discovery.type: single-node
+    restart: always
     networks:
       - elk
 
@@ -41,12 +42,13 @@ services:
         target: /usr/share/logstash/pipeline
         read_only: true
     ports:
-      - "15044:5044"
-      - "15000:5000/tcp"
-      - "15000:5000/udp"
-      - "19600:9600"
+      - "$LOGSTASH_PORT1:5044"
+      - "$LOGSTASH_PORT2:5000/tcp"
+      - "$LOGSTASH_PORT3:5000/udp"
+      - "$LOGSTASH_PORT4:9600"
     environment:
       LS_JAVA_OPTS: "-Xmx512m -Xms512m"
+    restart: always
     networks:
       - elk
     depends_on:
@@ -63,7 +65,8 @@ services:
         target: /usr/share/kibana/config/kibana.yml
         read_only: true
     ports:
-      - "15601:5601"
+      - "$KIBANA_PORT:5601"
+    restart: always
     networks:
       - elk
     depends_on:

init-kibana.sh

+#!/bin/bash
+
+. util.sh
+. .env
+
+kibana_host="localhost:$KIBANA_PORT"
+es_host="localhost:$ES_PORT1"
+
+function setDefaultIndex() {
+  curl --location --request POST "http://$kibana_host/api/kibana/settings" \
+    --header 'kbn-version: 7.12.0' \
+    --header 'Content-Type: application/json' \
+    --data-raw "{\"changes\":{\"defaultIndex\":\"$1\"}}"
+}
+
+function addILMPolicy() {
+  resp=$(
+    curl -s -XPUT "http://$es_host/_ilm/policy/service-log" \
+      --header 'Content-Type: application/json' \
+      --data-raw '{"policy":{"phases":{"hot":{"min_age":"0ms","actions":{"set_priority":{"priority":100}}},"delete":{"min_age":"365d","actions":{"delete":{"delete_searchable_snapshot":true}}}}}}'
+  )
+  echo "$resp"
+  if contain "$resp" '{"acknowledged":true}'; then
+    green "index life cycle import success"
+  else
+    red "index life cycle import failed"
+  fi
+}
+
+function addIndexTemplate() {
+  resp=$(
+    curl -s -XPUT "http://$es_host/_index_template/service-log" \
+      --header 'Content-Type: application/json' \
+      --data-raw '{"template":{"settings":{"index":{"lifecycle":{"name":"service-log"}}},"mappings":{"_source":{"excludes":[],"includes":[],"enabled":true},"_routing":{"required":false},"dynamic":true,"numeric_detection":false,"date_detection":true,"dynamic_date_formats":["strict_date_optional_time","yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"],"dynamic_templates":[]}},"index_patterns":["*-service-*"]}'
+  )
+  echo "$resp"
+  if contain "$resp" '{"acknowledged":true}'; then
+    green "index template import success"
+  else
+    red "index template import failed"
+  fi
+}
+
+function uploadConfig() {
+  resp=$(curl -s -XPOST "$kibana_host/api/saved_objects/_import?overwrite=true" --form "file=@kibana/export.ndjson" --header 'kbn-xsrf: true')
+  echo "$resp"
+  if contain "$resp" 'success'; then
+    green "config upload succeed"
+  else
+    red "config upload failed"
+  fi
+}
+
+function notHealth() {
+  resp=$(curl -sIL -m 5 -w "%{http_code}" -o /dev/null -XGET "$kibana_host/api/features" --header 'kbn-xsrf: true')
+  if [ "$resp" == 200 ]; then
+    return 1
+  else
+    return 0
+  fi
+}
+
+start=$(date +%s)
+while notHealth; do
+  end=$(date +%s)
+  take=$((end - start))
+  red "Kibana not ready yet, waiting... time: $take"
+  sleep 1s
+done
+green "======================================="
+green "Kibana is ready. Initialization begins."
+green "======================================="
+addILMPolicy
+addIndexTemplate
+uploadConfig
+green ""
+green "========================="
+green "Initialization completed."
+green "========================="
+blue "Visit the following website to discover data"
+blue "http://$kibana_host/app/discover"
+green "End"

kibana/export.ndjson

+{"attributes":{"buildNum":39309,"dateFormat":"YYYY-MM-DD HH:mm:ss.SSS","dateFormat:dow":"Monday","dateFormat:tz":"Asia/Shanghai","dateNanosFormat":"YYYY-MM-DD HH:mm:ss.SSSSSSSSS","defaultColumns":["message"],"defaultIndex":"3dd09b40-b302-11eb-8677-fd4740a7c42a"},"coreMigrationVersion":"7.12.0","id":"7.12.0","migrationVersion":{"config":"7.12.0"},"references":[],"type":"config","updated_at":"2021-07-02T05:10:07.453Z","version":"Wzc3NjUxLDExXQ=="}
+{"attributes":{"fieldAttrs":"{}","fields":"[]","runtimeFieldMap":"{}","timeFieldName":"@timestamp","title":"role-service*"},"coreMigrationVersion":"7.12.0","id":"a26fc570-b6d7-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-05-17T06:18:02.439Z","version":"WzYwMjE4LDExXQ=="}
+{"attributes":{"fieldAttrs":"{}","fields":"[]","runtimeFieldMap":"{}","timeFieldName":"@timestamp","title":"notify-service-v1*"},"coreMigrationVersion":"7.12.0","id":"cf0150c0-bf9e-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-05-28T10:23:56.364Z","version":"WzY0Mjc1LDExXQ=="}
+{"attributes":{"fieldAttrs":"{\"service.version\":{\"count\":2}}","fields":"[]","runtimeFieldMap":"{}","timeFieldName":"@timestamp","title":"others*"},"coreMigrationVersion":"7.12.0","id":"cb829560-b3d9-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-05-31T02:53:05.046Z","version":"WzcxMjI4LDExXQ=="}
+{"attributes":{"fieldAttrs":"{\"json.content\":{\"count\":13},\"json.duration\":{\"count\":7},\"json.level\":{\"count\":5},\"json.span\":{\"count\":4},\"json.trace\":{\"count\":4},\"log.file.path\":{\"count\":4},\"message\":{\"count\":10},\"service.name\":{\"count\":1},\"service.version\":{\"count\":2}}","fields":"[]","runtimeFieldMap":"{}","timeFieldName":"@timestamp","title":"*-service*"},"coreMigrationVersion":"7.12.0","id":"3dd09b40-b302-11eb-8677-fd4740a7c42a","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-06-22T02:40:13.017Z","version":"Wzc1ODg4LDExXQ=="}
+{"attributes":{"fieldAttrs":"{\"json.content\":{\"count\":2},\"message\":{\"count\":11},\"_id\":{\"count\":8},\"_type\":{\"count\":1},\"json.level\":{\"count\":6}}","fields":"[]","runtimeFieldMap":"{}","timeFieldName":"@timestamp","title":"trace-service-v2*"},"coreMigrationVersion":"7.12.0","id":"d2394ae0-c1ba-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-06-30T07:33:22.471Z","version":"Wzc2OTk3LDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"auth-service-v1*"},"coreMigrationVersion":"7.12.0","id":"66aa7500-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:20.432Z","version":"Wzc3NTQ2LDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"*-service-v1*"},"coreMigrationVersion":"7.12.0","id":"65763ac0-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:18.412Z","version":"Wzc3NTQyLDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"*-service-v2*"},"coreMigrationVersion":"7.12.0","id":"661030d0-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:19.421Z","version":"Wzc3NTQ0LDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"department-service-v2*"},"coreMigrationVersion":"7.12.0","id":"67441cf0-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:21.439Z","version":"Wzc3NTQ4LDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"gateway-service-v2*"},"coreMigrationVersion":"7.12.0","id":"69105170-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:24.455Z","version":"Wzc3NTU0LDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"gateway-service-v1*"},"coreMigrationVersion":"7.12.0","id":"687c9cf0-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:23.487Z","version":"Wzc3NTUyLDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"process-service-v2*"},"coreMigrationVersion":"7.12.0","id":"6c104b00-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:29.489Z","version":"Wzc3NTY0LDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"file-service-v1*"},"coreMigrationVersion":"7.12.0","id":"67dc8c60-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:22.438Z","version":"Wzc3NTUwLDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"pay-service-v1*"},"coreMigrationVersion":"7.12.0","id":"6b7606d0-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:28.477Z","version":"Wzc3NTYyLDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"proof-service-v1*"},"coreMigrationVersion":"7.12.0","id":"6ca9cbe0-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:30.494Z","version":"Wzc3NTY2LDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"swagger-service-v2*"},"coreMigrationVersion":"7.12.0","id":"6faef590-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:35.561Z","version":"Wzc3NTc0LDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"template-service-v2*"},"coreMigrationVersion":"7.12.0","id":"704198a0-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:36.522Z","version":"Wzc3NTc2LDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"swagger-service-v1*"},"coreMigrationVersion":"7.12.0","id":"6f14ff80-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:34.552Z","version":"Wzc3NTcyLDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"tx-service-v1*"},"coreMigrationVersion":"7.12.0","id":"7173d710-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:38.529Z","version":"Wzc3NTgwLDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"user-service-v1*"},"coreMigrationVersion":"7.12.0","id":"720e4250-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:39.541Z","version":"Wzc3NTgyLDExXQ=="}
+{"attributes":{"timeFieldName":"@timestamp","title":"role-service-v2*"},"coreMigrationVersion":"7.12.0","id":"6e7e8be0-daf2-11eb-bdce-75a994460910","migrationVersion":{"index-pattern":"7.11.0"},"references":[],"type":"index-pattern","updated_at":"2021-07-02T05:00:33.566Z","version":"Wzc3NTcwLDExXQ=="}
+{"attributes":{"columns":["json.content","json.duration","json.level","json.span","json.trace"],"description":"","hits":0,"kibanaSavedObjectMeta":{"searchSourceJSON":"{\"query\":{\"query\":\"\",\"language\":\"kuery\"},\"filter\":[{\"meta\":{\"alias\":null,\"negate\":false,\"disabled\":false,\"type\":\"exists\",\"key\":\"json.content\",\"value\":\"exists\",\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index\"},\"exists\":{\"field\":\"json.content\"},\"$state\":{\"store\":\"appState\"}}],\"indexRefName\":\"kibanaSavedObjectMeta.searchSourceJSON.index\"}"},"sort":[["@timestamp","desc"]],"title":"json","version":1},"coreMigrationVersion":"7.12.0","id":"df93aa50-b30f-11eb-8677-fd4740a7c42a","migrationVersion":{"search":"7.9.3"},"references":[{"id":"3dd09b40-b302-11eb-8677-fd4740a7c42a","name":"kibanaSavedObjectMeta.searchSourceJSON.index","type":"index-pattern"},{"id":"3dd09b40-b302-11eb-8677-fd4740a7c42a","name":"kibanaSavedObjectMeta.searchSourceJSON.filter[0].meta.index","type":"index-pattern"}],"type":"search","updated_at":"2021-05-12T10:50:32.181Z","version":"WzU1NTYxLDExXQ=="}
+{"exportedCount":23,"missingRefCount":0,"missingReferences":[]}
\ No newline at end of file

logstash/pipeline/logstash.conf

@@ -60,6 +60,12 @@ filter {
                 mutate { add_field => { "[service][name]" => "template-service" } }
         }else if "trace-service" in [log][file][path] {
                 mutate { add_field => { "[service][name]" => "trace-service" } }
+        }else if "sync" in [log][file][path] {
+                mutate { add_field => { "[service][name]" => "sync" } }
+        }else if "convert" in [log][file][path] {
+                mutate { add_field => { "[service][name]" => "convert" } }
+        }else if "exrpc" in [log][file][path] {
+                mutate { add_field => { "[service][name]" => "exrpc" } }
         }else {
                 mutate { add_field => { "[service][name]" => "others" } }
         }

start-and-init.sh

+#!/bin/bash
+
+. start.sh
+. init-kibana.sh
\ No newline at end of file

start.sh

+#!/bin/bash
+
+. ./util.sh
+. ./.env
+
+function checkPort(){
+  if [ "$(sudo netstat -nlput | grep -c $1)" -gt 0 ]; then
+    rad "==========================================================="
+    rad "The port $1 is already in use, change it in file \".env\"."
+    rad "==========================================================="
+    exit
+  fi
+}
+
+function checkAllPort() {
+  checkPort "$ES_PORT1"
+  checkPort "$ES_PORT2"
+  checkPort "$LOGSTASH_PORT1"
+  checkPort "$LOGSTASH_PORT2"
+  checkPort "$LOGSTASH_PORT3"
+  checkPort "$LOGSTASH_PORT4"
+  checkPort "$KIBANA_PORT"
+}
+
+function start(){
+  docker-compose up -d
+  green "==============="
+  green "start succeed"
+  green "==============="
+}
+
+function checkPS(){
+  if [ "$(docker ps | grep -c 'docker-elk' )" -gt 0 ]; then
+    red "======================="
+    red "ELK are already started"
+    red "======================="
+    exit
+  fi
+}
+
+checkPS
+checkAllPort
+start
\ No newline at end of file

util.sh

+#!/bin/bash
+blue(){
+    echo -e "\033[34m\033[01m$1\033[0m"
+}
+green(){
+    echo -e "\033[32m\033[01m$1\033[0m"
+}
+red(){
+    echo -e "\033[31m\033[01m$1\033[0m"
+}
+
+if [[ -f /etc/redhat-release ]]; then
+    release="centos"
+    systemPackage="yum"
+    systempwd="/usr/lib/systemd/system/"
+elif cat /etc/issue | grep -Eqi "debian"; then
+    release="debian"
+    systemPackage="apt-get"
+    systempwd="/lib/systemd/system/"
+elif cat /etc/issue | grep -Eqi "ubuntu"; then
+    release="ubuntu"
+    systemPackage="apt-get"
+    systempwd="/lib/systemd/system/"
+elif cat /etc/issue | grep -Eqi "centos|red hat|redhat"; then
+    release="centos"
+    systemPackage="yum"
+    systempwd="/usr/lib/systemd/system/"
+elif cat /proc/version | grep -Eqi "debian"; then
+    release="debian"
+    systemPackage="apt-get"
+    systempwd="/lib/systemd/system/"
+elif cat /proc/version | grep -Eqi "ubuntu"; then
+    release="ubuntu"
+    systemPackage="apt-get"
+    systempwd="/lib/systemd/system/"
+elif cat /proc/version | grep -Eqi "centos|red hat|redhat"; then
+    release="centos"
+    systemPackage="yum"
+    systempwd="/usr/lib/systemd/system/"
+fi
+
+function contain() {
+  if [ "$(echo "$1" | grep -c "$2")" -gt 0 ]; then return 0; else return 1; fi
+}