Commit Graph

64 Commits

Author SHA1 Message Date
Antoine Cotten
5dd178ac1a chore(curator): update to 8.0.10 2024-02-12 13:46:14 +01:00
Antoine Cotten
bd55cfaf64 feat(fleet): collect Docker logs and metrics
Mirrors the behaviour of the Metricsbeat and Filebeat extensions through
Elastic Agent, using the Docker integration.
2024-02-12 12:36:56 +01:00
Antoine Cotten
44822f6045 fix(filebeat): set explicit Docker-centric defaults
The default_config changed in elastic/beats@41ab08cd (v8.12.0) from
Docker to Kubernetes defaults. This change also changes the default
output from 'container' to 'input'.

Before we are able to update to v8.12, we need to set defaults which are
suitable for the configuration that docker-elk ships with.

Refs:
  https://github.com/elastic/beats/blob/v8.12.0/filebeat/autodiscover/builder/hints/config.go#L30-L51
  https://www.elastic.co/guide/en/beats/filebeat/8.12/filebeat-input-container.html
  https://www.elastic.co/guide/en/beats/filebeat/8.12/filebeat-input-filestream.html
2024-02-12 10:00:07 +01:00
Antoine Cotten
39510e0c0c doc: Convert deprecated Alerts syntax markers
https://docs.github.com/en/get-started/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax#alerts

Signed-off-by: Antoine Cotten <hello@acotten.com>
2023-10-17 10:26:25 +02:00
Antoine Cotten
6db3e37334 feat(ext/curator): Update to v8
Curator is now major version locked with Elasticsearch.
https://www.elastic.co/guide/en/elasticsearch/client/curator/current/versions.html
2023-02-17 16:06:30 +01:00
docker-elk-updater[bot]
717966a2be Set FLEET_SERVER_HOST explicitly 2023-02-01 00:08:22 +00:00
Antoine Cotten
0ebda1156f doc(ext): Clarify purpose of exposed TCP ports 2022-11-22 14:48:12 +01:00
Antoine Cotten
e202d6ba7c doc(ext/entsearch): Simplify encryption key example 2022-11-22 14:33:37 +01:00
Antoine Cotten
87f5894f65 feat(ext/beats): Use unprivileged publish users 2022-11-22 11:37:50 +01:00
Antoine Cotten
141b38a083 feat(ext/metricbeat): Further drop perms for monitoring
The built-in 'remote_monitoring_user' has full permissions on monitoring
indices, which is more than Metricbeat needs to collect metrics from
remote components.
2022-11-18 21:47:46 +01:00
Antoine Cotten
4961f99438 feat(ext/metricbeat): Add switch to disable monit modules 2022-11-18 19:20:46 +01:00
Antoine Cotten
654a18dcdb feat(ext/beats): Monitor using unprivileged user
Both the collection and storage of monitoring data are now performed
using the unpriviliged built-in users 'remote_monitoring_user' and
'beats_system' respectively, instead of the 'elastic' superuser.
2022-11-18 17:10:20 +01:00
Antoine Cotten
595f42cc90 doc: Fix markdown linter warnings 2022-11-18 16:06:10 +01:00
Antoine Cotten
00e1964a2f doc(ext/metricbeat): Add screenshots 2022-11-18 14:01:02 +01:00
Antoine Cotten
436f7076d7 doc(ext/fleet): Add screenshots 2022-11-18 14:01:02 +01:00
Antoine Cotten
ac464b2d99 feat(ext/beats): Monitor via internal collectors 2022-11-18 14:01:02 +01:00
Antoine Cotten
def0aa6665 feat(ext/metricbeat): Enable monitoring of Kibana 2022-11-18 14:01:02 +01:00
Antoine Cotten
a150f6dc99 feat(ext/metricbeat): Enable monitoring of Logstash 2022-11-18 12:05:18 +01:00
Antoine Cotten
4f4767cb5a feat(ext/metricbeat): Enable stack monitoring by default 2022-11-17 21:44:55 +01:00
Antoine Cotten
9f674eca40 chore: Clean up config 2022-11-17 21:24:46 +01:00
Antoine Cotten
32bf9423ed feat(ext/fleet): Add example of APM Server agent 2022-11-16 14:39:22 +01:00
Antoine Cotten
1943f25282 feat(ext/fleet): Store state in a volume
Prevents duplicate enrollments when the fleet-server container is
re-created.
2022-11-15 15:48:35 +01:00
Antoine Cotten
7d95a3f7fa chore(ext/fleet): Delegate Kibana init to Elastic Agent
Ref. https://github.com/elastic/elastic-agent/blob/v8.5.0/internal/pkg/agent/cmd/container.go#L105-L116
2022-11-15 14:39:20 +01:00
Antoine Cotten
58613c2dcb fix(ext/fleet): Restart on failure
It is expected that Elastic Agent fails when Kibana is not ready. To
simplify the ergonomics of this extension, and avoid having to wait
before starting it, restart the Fleet Server service (container)
automatically on failures.

Closes #763
2022-10-19 18:09:59 +02:00
Antoine Cotten
30cb8cb484 feat: Add Fleet extension, remove standalone APM Server (#760)
The preferred way to run APM Server is via the APM integration of the
Elastic Agent.

Ref. https://www.elastic.co/guide/en/apm/guide/8.4/legacy-apm-overview.html
2022-10-06 11:58:02 +02:00
Antoine Cotten
b11cc1cd7c fix(doc): Update branch in links to Elastic repos 2022-09-20 16:22:57 +02:00
docker-elk-updater[bot]
5b939e457e Update to v8.4.0 2022-08-25 09:33:50 +00:00
Evan
b6eb747a4a fix: Logstash TCP input uses IANA registered port 5000 2022-07-17 00:00:38 +02:00
Antoine Cotten
29df29f808 feat(ext): Add Heartbeat extension
Closes #672
2022-06-03 17:33:15 +02:00
Antoine Cotten
1557e27bff doc: Render notes and warnings using GitHub admonitions
https://github.com/github/feedback/discussions/16925
2022-05-21 12:14:56 +02:00
Antoine Cotten
4319be7cd0 fix: Revert removal of -Xms JVM option
Logstash has '-Xms1g' set inside 'config/jvm.options', so either -Xms
must be set, or -Xmx must be greater than 1g (which is too high of a
default for docker-elk).

Elastic explicitly recommends setting -Xms and -Xmx to the same value
for Elasticsearch:
  https://www.elastic.co/guide/en/elasticsearch/reference/8.2/advanced-configuration.html#set-jvm-heap-size
2022-05-16 21:29:08 +02:00
Antoine Cotten
239099874a chore: Raise max heap size of Elasticsearch to 512 MB
This value should be a sane default. Modern versions require more
resources, and nowadays any consumer hardware should be able to have
access to this amount of memory.

Additionally, do not set a lower bound on the heap size, so that unused
space isn't unnecessarily committed to the JVM.
2022-05-16 20:53:17 +02:00
Antoine Cotten
924a1e5987 chore: Update Compose file version to 3.7
Allows enabling the 'init' option in the 'setup' service.

Requires Docker 18.06.0 and Compose 1.22.0.
2022-02-28 09:50:52 +01:00
Antoine Cotten
eb1c6fa27e chore: Rename ELK_VERSION to ELASTIC_VERSION
More appropriate name considering that extensions, which aren't part of
the "ELK" stack, also rely on this variable.
2022-02-28 09:10:30 +01:00
Antoine Cotten
9877b39900 Automate management of required roles and user passwords (#671) 2022-02-21 11:19:43 +01:00
Antoine Cotten
641290c20a Apply SELinux options to bind mounts
https://docs.docker.com/storage/bind-mounts/

Related to #639
Closes #661
2022-02-11 20:11:22 +01:00
Antoine Cotten
ca98168da1 Quote passwords to prevent interpreting leading special chars
Closes #667
2022-02-11 19:10:58 +01:00
Antoine Cotten
6704d9f1d7 Update to v8.0.0 (#544)
List of changes impacting docker-elk:

- [logstash]: The output to Elasticsearch is handled as a data stream.

  Starting with v8.0.0, the `elasticsearch` output for Logstash sends
  log data to a data stream instead of `logstash-*` indices by default.
  The name of the default data stream is `logs-generic-default`.
  docker-elk remains unopinionated and simply uses Elastic's defaults
  like it always has, so users who prefer to retain the old behaviour
  need to explicitly opt-out of data streams in their Logstash
  pipelines.

  Refs:
  - https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html
  - https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data-streams

- [logstash]: The (legacy) monitoring data collection is now disabled.

  This feature was deprecated since v7.9.0, and removed in v8.0.0.

  Ref: https://www.elastic.co/guide/en/logstash/current/monitoring-internal-collection-legacy.html

- [kibana]: An index pattern for `logs-*` indices is automatically
  created.

  It used to be required to manually create an index pattern for indices
  managed by Logstash, even when using the default Logstash indices.
  This is no longer the case since the output data is now being handled
  as a data stream, and Kibana automatically creates index patterns for
  these.

- [elasticsearch]: The command line tool `elasticsearch-setup-passwords`
  was deprecated in favour of a new `elasticsearch-reset-password` tool.

  Passwords for built-in users must now be generated one by one.

  Ref: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-passwords.html

- [enterprise-search]: Kibana is now the new management interface, and
  the only one available moving forward.

  The old standalone Enterprise Search interface was removed in v8.0.0.

  Ref: https://www.elastic.co/guide/en/enterprise-search/current/user-interfaces.html
2022-02-10 17:19:04 +01:00
Antoine Cotten
3882ce97e1 doc: Update links to Legacy APM Server Reference 2022-02-04 15:44:55 +01:00
Antoine Cotten
1815972838 Add Filebeat extension 2021-12-04 02:17:41 +01:00
Antoine Cotten
40e6f2744b metricbeat: Reference docs for current instead of master 2021-01-16 21:37:19 +01:00
Antoine Cotten
1273e41260 metricbeat: Improve README, add notes about config 2021-01-16 21:29:11 +01:00
Mustafa Guney
f592f221c4 Add Metricbeat extension (#567)
Co-authored-by: Antoine Cotten <hello@acotten.com>
2021-01-16 21:11:24 +01:00
Antoine Cotten
282a8bbc85 ci: Add test for Logspout extension 2021-01-07 19:53:50 +01:00
Antoine Cotten
94104bebb9 logspout: Sync build script with upstream 2020-12-23 14:16:10 +01:00
Antoine Cotten
ace6e71cff Enable linter for Markdown files 2020-10-18 20:32:23 +02:00
Antoine Cotten
b908c5b2ec Add Action to track dead links in docs 2020-10-18 18:49:57 +02:00
Antoine Cotten
95272b72f4 Simplify curator extension 2020-10-16 20:30:10 +02:00
Antoine Cotten
c0dcc2489b Update to v7.8.0 2020-06-30 10:06:40 +02:00
Antoine Cotten
155c33cc91 Rename App Search to Enterprise Search
Starting with 7.7, Elastic App Search and Elastic Workplace Search are
now available as a single download known as Elastic Enterprise Search.
2020-06-10 15:39:44 +02:00