Metricbeat

• Change cloud.provider from ec2 to aws and from gce to gcp in add_cloud_metadata to align with ECS. 10775 11687

Affecting all Beats

• Fix formatting for event.duration, "human readable" was not working well for this. 11675

• Fix initialization of the TCP input logger. 11605

Auditbeat

• Package dataset: Log error when Homebrew is not installed. 11667

Heartbeat

• Fix NPE on some monitor configuration errors. 11910

Metricbeat

• Change add_cloud_metadata processor to not overwrite cloud field when it already exist in the event. 11612 11305

Auditbeat

• Add support to the system package dataset for the SUSE OS family. 11634

Metricbeat

• Prevent the docker/memory metricset from processing invalid events before container start 11676

libbeat/docs/release-notes/7.0.0.asciidoc

Affecting all Beats

• Relax validation of the X-Pack license UID value. 11640

• Fix a parsing error with the X-Pack license check on 32-bit system. 11650

• Fix ILM policy always being overwritten. 11671

• Fix template always being overwritten. 11671

Auditbeat

• Package dataset: Nullify Librpm’s rpmsqEnable. 11628

Filebeat

• Fix add_docker_metadata source matching, using log.file.path field now. 11577

Auditbeat

• Process dataset: Only report processes with executable. 11232

• Shorten entity IDs. 11405

Metricbeat

• Add connection and request timeouts for HTTP helper. 11032

Affecting all Beats

• Fixed OS family classification in add_host_metadata for Amazon Linux, Raspbian, and RedHat Linux. 9134 11494

Auditbeat

• Package dataset: dlopen versioned librpm shared objects. 11565

Filebeat

• Don’t apply multiline rules in Logstash json logs. 11346

• Fix panic in add_kubernetes_metadata processor when key log does not exist. 11543 11549

• Fix goroutine leak happening when harvesters are dynamically stopped. 11263

Metricbeat

• Add _bucket to histogram metrics in Prometheus Collector 11578

Auditbeat

• Login dataset: Add event category and type. 11339

Filebeat

• Add support for MySQL 8.0, Percona 8.0 and MariaDB 10.3. 11417

Affecting all Beats

• On Google Cloud Engine (GCE) the add_cloud_metadata will now trim the project info from the cloud.machine.type and cloud.availability_zone. 10968

• Add cleanup_timeout option to docker autodiscover, to wait some time before removing configurations after a container is stopped. 10374 10905

• Empty meta.json file will be treated as a missing meta file. 8558

• Rename migration.enabled config to migration.6_to_7.enabled. 11284

• Initialize the Paths before the keystore and save the keystore into data/{beatname}.keystore. 10706

• Beats Xpack now checks for Basic license on connect. 11296

Auditbeat

• Process dataset: Only report processes with executable. 11232

Filebeat

• Set ecs: true in user_agent processors when loading pipelines with Filebeat 7.0.x into Elasticsearch 6.7.x. 10655 10875

Metricbeat

• Migrate docker module to ECS. 10927

Functionbeat

• Correctly extract Kinesis Data field from the Kinesis Record. 11141

Affecting all Beats

• Reconnections of Kubernetes watchers are now logged at debug level when they are harmless. 10988

• Add missing host.* fields to fields.yml. 11016

• Include ip and boolean type when generating index pattern. 10995

• Using an environment variable for the password when enrolling a beat will now raise an error if the variable doesn’t exist. 10936

• Cancelling enrollment of a beat will not enroll the beat. 10150

• Allow to configure Kafka fetching strategy for the topic metadata. 10682

Auditbeat

• Package: Disable librpm signal handlers. 10694

• Login: Handle different bad login UTMP types. 10865

• System module: Fix and unify bucket closing logic. 10897

• User dataset: Numerous fixes to error handling. 10942

Filebeat

• Fix errors in filebeat Zeek dashboard and README files. Add notice.log support. 10916

• Fix a bug when converting NetFlow fields to snake_case. 10950

• Add on_failure handler for Zeek ingest pipelines. Fix one field name error for notice and add an additional test case. 11004 11105

• Fix issue preventing docker container events to be stored if the container has a network interface without ip address. 11225 11247

• Change URLPATH grok pattern to support brackets. 11135 11252

• Add support for iis log with different address format. 11255 11256

Heartbeat

• Fix checks for TCP send/receive data 11118

Metricbeat

• Migrate docker autodiscover to ECS. 10757 10862

• Fix issue in kubernetes module preventing usage percentages to be properly calculated. 10946

• Fix for not reusable http client leading to connection leaks in Jolokia module 11014

• Fix parsing error using GET in Jolokia module. 11075 11071

• Collect metrics when EC2 instances are not in running state. 11008 11023

• Change ECS field cloud.provider to aws. 11023

• Add documentation about jolokia autodiscover fields. 10925 10979

• Add missing aws.ec2.instance.state.name into fields.yml. 11219 11221

• Fix ec2 metricset to collect metrics from Cloudwatch with the same timestamp. 11142

• Fix potential memory leak in stopped docker metricsets 11294

Packetbeat

• Avoid reporting unknown MongoDB opcodes more than once. 10878

Winlogbeat

• Prevent Winlogbeat from dropping events with invalid XML. 11006

• Fix Winlogbeat escaping CR, LF and TAB characters. 11328 11357

Functionbeat

Affecting all Beats

• Add ip fields to default_field in Elasticsearch template. 11035

Auditbeat

• Move System module to beta. 10800

Filebeat

• Add ISO8601 timestamp support in syslog metricset. 8716 10736

• Add support for loading custom NetFlow and IPFIX field definitions to netflow input. 10945 11223

• Added categorization fields for SSH login events in the system/auth fileset. 11334

Metricbeat

• Add filters and pie chart for AWS EC2 dashboard. 10596

Journalbeat

• Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).

Affecting all Beats

• Embedded html is not escaped anymore by default. 9914

• Remove port settings from Logstash and Redis output. 9934

• Rename process.exe to process.executable in add_process_metadata to align with ECS. 9949

• Import ECS change ecs#308: leaf field user.group is now the group field set. 10275

• Update the code of Central Management to align with the new returned format. 10019

• Docker and Kubernetes labels/annotations will be "dedoted" by default. 10338

• Remove --setup command line flag. 10138

• Remove --version command line flag. 10138

• Remove --configtest command line flag. 10138

• Move output.elasticsearch.ilm settings to setup.ilm. 10347

• ILM will be available by default if Elasticsearch > 7.0 is used. 10347

Auditbeat

• Rename process.exe to process.executable in auditd module to align with ECS. 9949

• Rename process.cwd to process.working_directory in auditd module to align with ECS. 10195

• Change data type of process.pid and process.ppid to number in JSON output of the auditd module. 10195

• Change data type of file.uid and file.gid to string in JSON output of the FIM module. 10195

• Field file.origin changed type from text to keyword. 10544

• Rename user fields to ECS in auditd module. 10456

• Rename event.type to auditd.message_type in auditd module because event.type is reserved for future use by ECS. 10536

• Rename auditd.messages to event.original and auditd.warnings to error.message. 10577

Filebeat

• Rename many kibana.log.* fields to map to ECS. 9301

• Modify apache/error dataset to follow ECS. 8963

• Rename many traefik.access.* fields to map to ECS. 9005

• Fix parsing of GC entries in elasticsearch server log. 9513 9810

• Rename read_timestamp to event.created for Redis input. 9924

• Rename a few elasticsearch.audit.* fields to map to ECS. 9293

• Rename read_timestamp to event.created for all Filebeat modules using it. 10139

• Rename many iis.error.* fields to map to ECS. 9955

• Adjust fileset haproxy.log to map to ECS. 10143

• Rename a few logstash.* fields to map to ECS, remove logstash.slowlog.message. 9935

• Rename a few mongodb.* fields to map to ECS. 10009

• Rename a few mysql.* fields to map to ECS. 10008

• Rename a few nginx.error.* fields to map to ECS. 10007

• Rename many auditd.log.* fields to map to ECS. 10192

• Filesets with multiple ingest pipelines added in 8914 only work with Elasticsearch >= 6.5.0 10001

• Remove service.name from Elastcsearch module. Replace by service.type. 10042

• Remove numeric coercions for user.id and group.id. IDs should be keyword. 10233

• Add grok pattern to support redis 5.0.3 log timestamp. 9819 10033

• Now save the 'first seen' timestamp in event.created (previously read_timestamp), instead of saving the parsed date. Now aligned with event.created semantics elsewhere. 10139

• Rename mysql.error.thread_id and mysql.slowlog.id to mysql.thread_id. 10161

• Remove mysql.error.timestamp and mysql.slowlog.timestamp. 10161

• Migrate multiple fields to event.duration, from modules "apache", "elasticsearch", "haproxy", "iis", "kibana", "mysql", "nginx", "postgresql" and "traefik", including http.response.elapsed_time (ECS). 10188, 10274

• Rename multiple fields to http.response.body.bytes, from modules "apache", "iis", "kibana", "nginx" and "traefik", including http.response.content_length (ECS). 10188

• Change type from haproxy.log fileset fields from text to keyword: response.captured_headers, request.captured_headers, raw_request_line, mode. 10397

• Change type of field backend_url and frontend_name in traefik.access metricset to type keyword. 10401

• Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above 10352

• Migrate Elasticsearch audit logs fields to ECS 10352

• Several text fields in the Logstash module are now indexed as keyword fields with text multi-fields (ECS). 10417

• Several text fields in the Elasticsearch module are now indexed as keyword fields with text multi-fields (ECS). 10414

• Move dissect pattern for traefik.access fileset from Filbeat to Elasticsearch. 10442

• The elasticsearch/deprecation fileset now indexes the component field under elasticsearch instead of elasticsearch.server. 10445

• Remove field kafka.log.trace.full from kafka.log fielset. 10398

• Change field kafka.log.class for kafka.log fileset from text to keyword. 10398

• Address add_kubernetes_metadata processor issue where old source field is still used for matcher. 10505 10506

• Change type of haproxy.source from text to keyword. 10506

• Rename event.type to suricata.eve.event_type in Suricata module because event.type is reserved for future use by ECS. 10575

• Populate more ECS fields in the Suricata module. 10006

• Rename setting filebeat.registry_flush to filebeat.registry.flush. 10504

• Rename setting filebeat.registry_file_permission to filebeat.registry.file_permission. 10504

• Remove setting filebeat.registry_file in favor of filebeat.registry.path. The registry file will be stored in a sub-directory by now. 10504

Heartbeat

• Remove monitor generator script that was rarely used. 9648

• monitor IDs are now configurable. Auto generated monitor IDs now use a different formula based on a hash of their config values. If you wish to have continuity with the old format of monitor IDs you’ll need to set the id property explicitly. 9697

• A number of fields have been aliased to their relevant counterparts in the url.* field. Existing visualizations should mostly work. The fields that have been moved are monitor.scheme → url.scheme, monitor.host → url.domain, resolve.host → url.domain, http.url → url.full, tcp.port → url.port. In addition to these moves the new fields url.username, url.password, url.path, and url.query are now present. It should be noted that the url.password field does not contain actual password values, but rather the text <hidden> 9570.

• The included Kibana HTTP dashboard is now removed in favor of the Uptime app in Kibana. 10294

Journalbeat

• Rename read_timestamp to event.created to align with ECS. 10043, 10139

• Rename host.name to host.hostname to align with ECS. 10043

• Fix typo in the field name container.id_truncated. 10525

• Rename container.image.tag to container.log.tag. 10561

• Change type of text fields to keyword. 10542

Metricbeat

• Migrate system process metricset fields to ECS. 10332

• Refactor Prometheus metric mappings 9948

• Removed Prometheus stats metricset in favor of just using Prometheus collector 9948

• Migrate system socket metricset fields to ECS. 10339

• Renamed direction values in sockets to ECS recommendations, from incoming/outcoming to inbound/outbound. 10339

• Adjust Redis.info metricset fields to ECS. 10319

• Change type of field docker.container.ip_addresses to ip instead of keyword. 10364

• Rename http.request.body field to http.request.body.content. 10315

• Adjust php_fpm.process metricset fields to ECS. 10366

• Adjust mongodb.status metricset to to ECS. 10368

• Refactor munin module to collect an event per plugin and to have more strict field mappings. namespace option has been removed, and will be replaced by service.name. 10322

• Change the following fields from type text to keyword: 10318

• ceph.osd_df.name

• ceph.osd_tree.name

• ceph.osd_tree.children

• kafka.consumergroup.meta

• kibana.stats.name

• mongodb.metrics.replication.executor.network_interface

• php_fpm.process.request_uri

• php_fpm.process.script

• Add service.name option to all modules to explicitly set service.name if it is unset. 10427

• Update a few elasticsearch.* fields to map to ECS. 10350

• Update a few logstash.* fields to map to ECS. 10350

• Update a few kibana.* fields to map to ECS. 10350

• Update rabbitmq.* fields to map to ECS. 10563

• Update haproxy.* fields to map to ECS. 10558 10568

• Collect all EC2 meta data from all instances in all states. 10628

• Fix MongoDB dashboard that had some incorrect field names from status Metricset 9795 9715

Packetbeat

• Adjust Packetbeat http fields to ECS Beta 2 9645

• http.request.body moves to http.request.body.content

• http.response.body moves to http.response.body.content

• Changed Packetbeat fields to align with ECS. 7968

• Removed trailing dot from domain names reported by the DNS protocol. 9941

Winlogbeat

• Adjust Winlogbeat fields to map to ECS. 10333

Functionbeat

• Correctly normalize Cloudformation resource name. 10087

• Functionbeat can now deploy a function for Kinesis. {10116}10116[10116]

• Allow functionbeat to use the keystore. 9009

Affecting all Beats

• Fix config appender registration. 9873

• Gracefully handle TLS options when enrolling a Beat. 9129

• The backing off now implements jitter to better distribute the load. 10172

• Fix TLS certificate DoS vulnerability. 10302

• Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289

• Fix encoding of timestamps when using disk spool. 10099

• Fix stopping of modules started by kubernetes autodiscover. 10476

• Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587

• Fix unauthorized error when loading dashboards by adding username and password into kibana config. 10513 10675

• Fix exclude_labels when there are dotted keys 10154

• Fix registry handle leak on Windows (https://github.com/elastic/go-sysinfo/pull/33). 9920

Auditbeat

• Enable System module config on Windows. 10237

Filebeat

• Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869, access log: 9955.

• Support haproxy log lines without captured headers. 9463 9958

• Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135

• Fix bad bytes count in docker input when filtering by stream. 10211

• Fixed data types for roles and indices fields in elasticsearch/audit fileset 10307

• Ensure source.address is always populated by the nginx module (ECS). 10418

• Support mysql 5.7.22 slowlog starting with time information. 7892 9647

Heartbeat

• Made monitors.d configuration part of the default config. 9004

• Fixed rare issue where TLS connections to endpoints with x509 certificates missing either notBefore or notAfter would cause the check to fail with a stacktrace. 9566

Journalbeat

• Do not stop collecting events when journal entries change. 9994

Metricbeat

• Fix panics in vsphere module when certain values where not returned by the API. 9784

• Fix pod UID metadata enrichment in Kubernetes module. 10081

• Fix issue that would prevent collection of processes without command line on Windows. 10196

• Fixed data type for tags field in docker/container metricset 10307

• Fixed data type for tags field in docker/image metricset 10307

• Fixed data type for isr field in kafka/partition metricset 10307

• Fixed data types for various hosts fields in mongodb/replstatus metricset 10307

• Added function to close sql database connection. 10355

• Fix issue with elasticsearch/node_stats metricset (x-pack) not indexing source_node field. 10639

Packetbeat

• Fix DHCPv4 dashboard that wouldn’t load in Kibana. 9850

• Fixed a crash when using af_packet capture 10477

Winlogbeat

• Close handle on signalEvent. 9838

Functionbeat

• Ensure that functionbeat is logging at info level not debug. 10262

• Add the required permissions to the role when deployment SQS functions. 9152

Affecting all Beats

• Update field definitions for http to ECS Beta 2 9645

• Add agent.id and agent.ephemeral_id fields to all beats. 9404

• Add name config option to add_host_metadata processor. 9943

• Add add_labels and add_tags processors. 9973

• Add missing file encoding to readers. 10080

• Introduce migration.enabled configuration. 9805

• Add alias field support in Kibana index pattern. 10075

• Add add_fields processor. 10119

• Add Kibana field formatter to bytes fields. 10184

• Document a few more auditd.log.* fields. 10192

• Support Kafka 2.1.0. 10440

• Add ILM mode auto to setup.ilm.enabled setting. This new default value detects if ILM is available 10347

• Add support to read ILM policy from external JSON file. 10347

• Add overwrite and check_exists settings to ILM support. 10347

• Generate Kibana index pattern on demand instead of using a local file. 10478

• Calls to Elasticsearch X-Pack APIs made by Beats won’t cause deprecation logs in Elasticsearch logs. {9656}9656[9656]

• Allow to unenroll a Beat from the UI. 9452

• Release Jolokia autodiscover as GA. 9706

• Allow Central Management to send events back to kibana. 9382

Auditbeat

• Add system module. 9546

• Add user.id (UID) and user.name for ECS. 10195

• Add group.id (GID) and group.name for ECS. 10195

• System module process dataset: Add user information to processes. 9963

• Add system package dataset. 10225

• Add system module login dataset. 9327

• Add entity_id fields. 10500

• Add seven dashboards for the system module. 10511

Filebeat

• Add convert_timezone option to Elasticsearch module to convert dates to UTC. 9756 9761

• Added module for parsing Google Santa logs. 9540

• Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399

• Add option to modules.yml file to indicate that a module has been moved 9432.

• Add support for ssl_request_log in apache2 module. 8088 9833

• Add support for iis 7.5 log format. 9753 9967

• Add service.type field to all Modules. By default the field is set with the module name. It can be overwritten with service.type config. 10042

• Add support for MariaDB in the slowlog fileset of mysql module. 9731

• Apache module’s error fileset now performs GeoIP lookup, like the access fileset. 10273

• Elasticsearch module’s slowlog now populates event.duration (ECS). 9293

• HAProxy module now populates event.duration and http.response.bytes (ECS). 10143

• Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137

• Add convert_timezone to nginx module. 9839 10148

• Add support for Percona in the slowlog fileset of mysql module. 6665 10227

• Added support for ingesting structured Elasticsearch audit logs 10352

• Added support for ingesting structured Elasticsearch slow logs 10445

• Added support for ingesting structured Elasticsearch deprecation logs 10445

• New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176

• Added support for ingesting structured Elasticsearch server logs 10428

• Populate more ECS fields in the Suricata module. 10006

• Add module zeek. 9931 10034

Heartbeat

• Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the docker key. 10258

Journalbeat

• Migrate registry from previously incorrect path. 10486

Metricbeat

• Add key metricset to the Redis module. 9582 9657 9746

• Add socket_summary metricset to system defaults, removing experimental tag and supporting Windows 9709

• Add docker event metricset. 9856

• Add 'performance' metricset to x-pack mssql module 9826

• Add DeDot for kubernetes labels and annotations. 9860 9939

• Add more meaningful metrics to 'performance' Metricset on 'MSSQL' module 10011

• Rename some fields in performance Metricset on MSSQL module to match the updated documentation from Microsoft 10074

• Add AWS EC2 module. 9257 9300

• Release windows Metricbeat module as GA. 10163

• Release traefik Metricbeat module as GA. 10166

• Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094

• List filesystems on Windows that have an access path but not an assigned letter 8916 10196

• Add nats module. 10071

• Release uswgi Metricbeat module GA. 10164

• Release php_fpm module as GA. 10198

• Release Memcached module as GA. 10199

• Release etcd module as GA. 10200

• Release Ceph module as GA. 10202

• Release aerospike module as GA. 10203

• Release kubernetes apiserver and event metricsets as GA 10212

• Release Couchbase module as GA. 10201

• Release RabbitMQ module GA. 10165

• Release envoyproxy module GA. 10223

• Release mongodb.metrics and mongodb.replstatus as GA. 10242

• Release mysql.galera_status as GA. 10242

• Release postgresql.statement as GA. 10242

• Release RabbitMQ Metricbeat module GA. 10165

• Release Dropwizard module as GA. 10240

• Release Graphite module as GA. 10240

• Release kvm module as beta. 10279

• Release http.server metricset as GA. 10240

• Release Nats module as GA. 10281

• Release munin module as GA. 10311

• Release Golang module as GA. 10312

• Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222

• Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261

• Rename 'db' Metricset to 'transaction_log' in MSSQL Metricbeat module 10109

• Add process arguments and the path to its executable file in the system process metricset 10332

• Added 'server' Metricset to Zookeeper Metricbeat module 8938 10341

• Release AWS module as GA. 10345

• Add overview dashboard to Zookeeper Metricbeat module 10379

Packetbeat

• Add network.community_id to Packetbeat flow events. 10061

• Add aliases for flow fields that were renamed. 7968 10063

• Add support to decode mysql prepare statement command. 8084

Functionbeat

• Mark Functionbeat as GA. 10564

Affecting all Beats

• Update add_cloud_metadata fields to adjust to ECS. 9265

• Automaticall cap signed integers to 63bits. 8991

• Rename beat.timezone to event.timezone. 9458

• Use _doc as document type. 9056

• Removed dashboards and index patterns generation for Kibana 5. 8927

• On systems with systemd, the Beats log is now written to journald by default rather than file. To revert this behaviour override BEAT_LOG_OPTS with an empty value. 8942.

Auditbeat

• Remove warning for deprecated option: "filters". 9002

Filebeat

• Allow beats to blacklist certain part of the configuration while using Central Management. 9099

• Remove warnings for deprecated options: "spool_size", "publish_async", "idle_timeout". 9002

• Rename many haproxy.* fields to map to ECS. 9117

• Rename many iis.access.* fields to map to ECS. 9084

• IIS module’s user agent string is no longer encoded (+ replaced with spaces). 9084

• Rename many system.syslog.* fields to map to ECS. 9135

• Rename many nginx.access.* fields to map to ECS. 9081

• Rename many system.auth.* fields to map to ECS. 9138

• Rename many apache2.access.* fields to map to ECS. 9245

• Rename apache2 module to apache. 9402

Metricbeat

• Allow beats to blacklist certain part of the configuration while using Central Management. 9099

• Remove warning for deprecated option: "filters". 9002

Packetbeat

• Renamed the flow event fields to follow Elastic Common Schema. 9121

• Renamed several client and server fields. IP, port, and process metadata are now contained under the client and server namespaces. 9303

Functionbeat

• The CLI will now log CloudFormation Stack events. 8912

• Function concurrency is now set to 5 instead of unreserved. 8992

Affecting all Beats

• Propagate Sync error when running SafeFileRotate. 9069

• Fix autodiscover configurations stopping when metadata is missing. 8851

• Log events at the debug level when dropped by encoding problems. 9251

• Refresh host metadata in add_host_metadata. 9359

• When collecting swap metrics for beats telemetry or system metricbeat module handle cases of free swap being bigger than total swap by assuming no swap is being used. 6271 9383

• Adding logging traces at debug level when the pipeline client receives the following events: onFilteredOut, onDroppedOnPublish. 9016

• Ignore non index fields in default_field for Elasticsearch. 9549

• Update Kibana index pattern attributes for objects that are disabled. 9644

• Enforce validation for the Central Management access token. 9621

• Update to Golang 1.11.4. 9627

Auditbeat

Filebeat

• Correctly parse December or Dec in the Syslog input. 9349

• Fix installation of haproxy dashboard. 9307 9313

• Don’t generate incomplete configurations when logs collection is disabled by hints. 9305

• Stop runners disabled by hints after previously being started. 9305

• Fix saved objects in filebeat haproxy dashboard. 9417

• Use log.source.address instead of log.source.ip for network input sources. 9487

• Rename many redis.log.* fields to map to ECS. 9315

• Rename many icinga.* fields to map to ECS. 9294

• Rename many postgresql.log.* fields to map to ECS. 9308

• Rename many kafka.log.* fields to map to ECS. 9297

• Add convert_timezone option to Logstash module to convert dates to UTC. 9756 9797

Metricbeat

• Fix issue preventing diskio metrics collection for idle disks. 9124 9125

• Fix panic on docker healthcheck collection on dockers without healthchecks. 9171

• Fix issue with not collecting Elasticsearch cross-cluster replication stats correctly. 9179

• The node.name field in the elasticsearch/node metricset now correctly reports the Elasticsarch node name. Previously this field was incorrectly reporting the node ID instead. 9209

Packetbeat

• Fix issue with process monitor associating traffic to the wrong process. 9151 9443

Affecting all Beats

• Unify dashboard exporter tools. 9097

• Add cache.ttl to add_host_metadata. 9359

• Add support for index lifecycle management (beta). 7963

• Always include Pod UID as part of Pod metadata. 9517

• Autodiscovery no longer requires that the condition field be set. If left unset all configs will be matched. 9029

• Add geo fields to add_host_metadata processor. 9392

Filebeat

• Added the redirect_stderr option that allows panics to be logged to log files. 8430

• Added detect_null_bytes selector to detect null bytes from a io.reader. 9210

• Added syslog_host variable to HAProxy module to allow syslog listener to bind to configured host. 9366

• Added support on Traefik for Common Log Format and Combined Log Format mixed which is the default Traefik format 8015 6111 8768.

• Add support for multi-core thread_id in postgresql module 9156 9482

Heartbeat

• Add last monitor status to dashboard table. Further break out monitors in dashboard table by monitor.ip. 9022

• Add central management support. 9254

Journalbeat

• Add cursor_seek_fallback option. 9234

Metricbeat

• Add settings to disable docker and cgroup cpu metrics per core. 9187 9194 9589

• The elasticsearch/node metricset now reports the Elasticsearch cluster UUID. 8771

• Add service.type field to Metricbeat. 8965

• Support GET requests in Jolokia module. 8566 9226

• Add freebsd support for the uptime metricset. 9413

• Add host.os.name field to add_host_metadata processor. 8948 9405

• Add more TCP statuses to socket_summary metricset. 9430

• Remove experimental tag from ceph metricsets. 9708

• Add MS SQL module to X-Pack https://github.com/elastic/beats/pull/9414[9414

Metricbeat

• event.duration is now in nano and not microseconds anymore. 8941

Affecting all Beats

• Dissect syntax change, use * instead of ? when working with field reference. 8054

Auditbeat

• Use initial_scan action for new paths. 7954

• Rename beat.name to agent.type, beat.hostname to agent.hostname, beat.version to agent.version.

• Rename source.hostname to source.domain in the auditd module. 9027

Filebeat

• Rename fileset.name to event.name. 8879

• Rename fileset.module to event.module. 8879

• Rename source to log.file.path and log.source.ip 8902

• Remove the deprecated prospector(s) option in the configuration use input(s) instead. 8909

• Rename offset to log.offset. 8923

• Rename source_ecs to source in the Filebeat Suricata module. 8983

Affecting all Beats

• Fixed -d CLI flag by trimming spaces from selectors. 7864

• Fixed Support add_docker_metadata in Windows by identifying systems' path separator. 7797

• Do not panic when no tokenizer string is configured for a dissect processor. 8895

• Start autodiscover consumers before producers. 7926

Filebeat

• Fixed a memory leak when harvesters are closed. 7820

• Fix improperly set config for CRI Flag in Docker Input 8899

• Just enabling the elasticsearch fileset and starting Filebeat no longer causes an error. 8891

• Fix macOS default log path for elasticsearch module based on homebrew paths. {pul}8939[8939]

Heartbeat

• Heartbeat now always downloads the entire body of HTTP endpoints, even if no checks against the body content are declared. This fixes an issue where timing metrics would be incorrect in scenarios where the body wasn’t used since the connection would be closed soon after the headers were sent, but before the entire body was. 8894

• Host header can now be overridden for HTTP requests sent by Heartbeat monitors. 9516

Metricbeat

• Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. 7789

• Add missing namespace field in http server metricset 7890

• Fix race condition when enriching events with kubernetes metadata. 9055 9067

Packetbeat

• Fixed the mysql missing transactions if monitoring a connection from the start. 8173

Affecting all Beats

• Add field host.os.kernel to the add_host_metadata processor and to the internal monitoring data. 7807

• Add debug check to logp.Logger 7965

• Count HTTP 429 responses in the elasticsearch output 8056

• Allow Bus to buffer events in case listeners are not configured. 8527

• Dissect will now flag event on parsing error. 8751

• add_cloud_metadata initialization is performed asynchronously to avoid delays on startup. 8845

• Add DeDot method in add_docker_metadata processor in libbeat. 9350 9505

Filebeat

• Make inputsource generic taking bufio.SplitFunc as input 7746

• Add custom unpack to log hints config to avoid env resolution 7710

• Make docker input check if container strings are empty 7960

• Keep unparsed user agent information in user_agent.original. 8537

• Allow to force CRI format parsing for better performance 8424

Heartbeat

• Add automatic config file reloading. 8023

Journalbeat

• Add the ability to check against JSON HTTP bodies with conditions. 8667

Metricbeat

• Add metrics about cache size to memcached module 7740

• Add experimental socket summary metricset to system module 6782

• Collect custom cluster display_name in elasticsearch/cluster_stats metricset. 8445

• Test etcd module with etcd 3.3. 9068

• All elasticsearch metricsets now have module-level cluster.id and cluster.name fields. 8770 8771 9164 9165 9166 9168

• All elasticsearch node-level metricsets now have node.id and node.name fields. 9168 9209

Packetbeat

• Add support to decode HTTP bodies compressed with gzip and deflate. 7915

• Added support to calculate certificates' fingerprints (MD5, SHA-1, SHA-256). 8180

• Support new TLS version negotiation introduced in TLS 1.3. 8647.

Affecting all Beats

• Relax validation of the X-Pack license UID value. 11640

• Fix a parsing error with the X-Pack license check on 32-bit system. 11650

• Fix OS family classification in add_host_metadata for Amazon Linux, Raspbian, and RedHat Linux. 9134 11494

• Fix false positives reported in the host.containerized field added by add_host_metadata. 11494

• Fix the add_host_metadata’s host.id field on older Linux versions. 11494

Auditbeat

• Package dataset: dlopen versioned librpm shared objects. 11565

• Package dataset: Nullify Librpm’s rpmsqEnable. 11628

Filebeat

• Don’t apply multiline rules in Logstash json logs. 11346

• Fix goroutine leak happening when harvesters are dynamically stopped. 11263

• Fix initialization of the TCP input logger. 11605

Metricbeat

• Prevent the docker/memory metricset from processing invalid events before container start 11676

Auditbeat

• Add support to the system package dataset for the SUSE OS family. 11634

Affecting all Beats

• Initialize the Paths before the keystore and save the keystore into data/{beatname}.keystore. 10706

Affecting all Beats

• Remove IP fields from default_field in Elasticsearch template. 11399

Affecting all Beats

• Port settings have been deprecated in redis/logstash output and will be removed in 7.0. 9915

• Update the code of Central Management to align with the new returned format. 10019

• Allow Central Management to send events back to kibana. 9382

• Fix panic if fields settting is used to configure hosts.x fields. 10824 10935

• Introduce query.default_field as part of the template. 11205

• Beats Xpack now checks for Basic license on connect. 11296

Filebeat

• Filesets with multiple ingest pipelines added in 8914 only work with Elasticsearch >= 6.5.0 10001

• Add grok pattern to support redis 5.0.3 log timestamp. 9819 10033

• Ingesting Elasticsearch audit logs is only supported with Elasticsearch 6.5.0 and above 8852

• Remove ecs option from user_agent processors when loading pipelines with Filebeat 6.7.x into Elasticsearch < 6.7.0. 10655 11362

Heartbeat

• Remove monitor generator script that was rarely used. 9648

Affecting all Beats

• Fix TLS certificate DoS vulnerability. 10303

• Fix panic and file unlock in spool on atomic operation (arm, x86-32). File lock was not released when panic occurs, leading to the beat deadlocking on startup. 10289

• Adding logging traces at debug level when the pipeline client receives the following events: onFilteredOut, onDroppedOnPublish. 9016

• Do not panic when no tokenizer string is configured for a dissect processor. 8895

• Fix a issue when remote and local configuration didn’t match when fetching configuration from Central Management. 10587

• Add ECS-like selectors and dedotting to docker autodiscover. 10757 10862

• Fix encoding of timestamps when using disk spool. 10099

• Include ip and boolean type when generating index pattern. 10995

• Using an environment variable for the password when enrolling a beat will now raise an error if the variable doesn’t exist. 10936

• Cancelling enrollment of a beat will not enroll the beat. 10150

• Remove IP fields from default_field in Elasticsearch template. 11399

Auditbeat

• Package: Disable librpm signal handlers. 10694

• Login: Handle different bad login UTMP types. 10865

• Fix hostname references in System module dashbords. 11064

• User dataset: Numerous fixes to error handling. 10942

Filebeat

• Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869 access log: 10029

• Fix bad bytes count in docker input when filtering by stream. 10211

• Fixed data types for roles and indices fields in elasticsearch/audit fileset 10307

• Cover empty request data, url and version in Apache2 modulehttps://github.com/elastic/beats/pull/10846[10846]

• Fix a bug with the convert_timezone option using the incorrect timezone field. 11055 11164

• Change URLPATH grok pattern to support brackets. 11135 11252

• Add support for iis log with different address format. 11255 11256

• Add fix to parse syslog message with priority value 0. 11010

Heartbeat

• Host header can now be overridden for HTTP requests sent by Heartbeat monitors. 9516

• Fix checks for TCP send/receive data 10777

Journalbeat

• Do not stop collecting events when journal entries change. 9994

Metricbeat

• Fix MongoDB dashboard that had some incorrect field names from status Metricset 9795 9715

• Fix issue that would prevent collection of processes without command line on Windows. 10196

• Fixed data type for tags field in docker/container metricset 10307

• Fixed data type for tags field in docker/image metricset 10307

• Fixed data type for isr field in kafka/partition metricset 10307

• Fixed data types for various hosts fields in mongodb/replstatus metricset 10307

• Added function to close sql database connection. 10355

• Fix parsing error using GET in Jolokia module. 11075 11071

Winlogbeat

• Fix Winlogbeat escaping CR, LF and TAB characters. 11328 11357

Functionbeat

• Correctly extract Kinesis Data field from the Kinesis Record. 11141

• Add the required permissions to the role when deployment SQS functions. 9152

Affecting all Beats

• Add ip fields to default_field in Elasticsearch template. 11035

• Add cleanup_timeout option to docker autodiscover, to wait some time before removing configurations after a container is stopped. 10374 10905

Auditbeat

• System module process dataset: Add user information to processes. 9963

• Add system package dataset. 10225

• Add system module login dataset. 9327

• Add entity_id fields. 10500

• Add seven dashboards for the system module. 10511

Filebeat

• Add field log.source.address and log.file.path to replace source. 9435

• Support mysql 5.7.22 slowlog starting with time information. 7892 9647

• Add support for ssl_request_log in apache2 module. 8088 9833

• Add support for iis 7.5 log format. 9753 9967

• Add support for MariaDB in the slowlog fileset of mysql module. 9731

• Add convert_timezone to nginx module. 9839 10148

• Add support for Percona in the slowlog fileset of mysql module. 6665 10227

• Added support for ingesting structured Elasticsearch audit logs 8852

• New iptables module that receives iptables/ip6tables logs over syslog or file. Supports Ubiquiti Firewall extensions. 8781 10176

• Populate more ECS fields in the Suricata module. 10006

Heartbeat

• Made monitors.d configuration part of the default config. 9004

• Autodiscover metadata is now included in events by default. So, if you are using the docker provider for instance, you’ll see the correct fields under the docker key. 10258

Metricbeat

• Add field event.dataset which is {module}.{metricset}.

• Add more TCP statuses to socket_summary metricset. 9430

• Remove experimental tag from ceph metricsets. 9708

• Add key metricset to the Redis module. 9582 9657

• Add DeDot for kubernetes labels and annotations. 9860 9939

• Add docker event metricset. 9856

• Release Ceph module as GA. 10202

• Release windows Metricbeat module as GA. 10163

• Release traefik Metricbeat module as GA. 10166

• List filesystems on Windows that have an access path but not an assigned letter 8916 10196

• Release uswgi Metricbeat module GA. 10164

• Release php_fpm module as GA. 10198

• Release Memcached module as GA. 10199

• Release etcd module as GA. 10200

• Release kubernetes apiserver and event metricsets as GA 10212

• Release Couchbase module as GA. 10201

• Release aerospike module as GA. 10203

• Release envoyproxy module GA. 10223

• Release mongodb.metrics and mongodb.replstatus as GA. 10242

• Release mysql.galera_status as Beta. 10242

• Release postgresql.statement as GA. 10242

• Release RabbitMQ Metricbeat module GA. 10165

• Release Dropwizard module as GA. 10240

• Release Graphite module as GA. 10240

• Release http.server metricset as GA. 10240

• Add support for MySQL 8.0 and tests also for Percona and MariaDB. 10261

• Release use of xpack.enabled: true flag in Elasticsearch and Kibana modules as GA. 10222

• Release Elastic stack modules (Elasticsearch, Logstash, and Kibana) as GA. 10094

• Add remaining memory metrics of pods in Kubernetes metricbeat module 10157

• Added 'server' Metricset to Zookeeper Metricbeat module 8938 10341

• Add overview dashboard to Zookeeper Metricbeat module 10379

Functionbeat

• Mark Functionbeat as GA. 10564

• Functionbeat can now deploy a function for Kinesis. 10116

• Allow functionbeat to use the keystore. 9009

Filebeat

• Deprecate field source. Will be replaced by log.source.address and log.file.path in 7.0. 9435

Metricbeat

• Deprecate field metricset.rtt. Replaced by event.duration which is in nano instead of micro seconds.

Packetbeat

• Support new TLS version negotiation introduced in TLS 1.3. 8647.

Journalbeat

• Journalbeat requires at least systemd v233 in order to follow entries after journal changes (rotation, vacuum).

Auditbeat

• System module: Fix and unify bucket closing logic. 10897

Filebeat

• Fix a bug when converting NetFlow fields to snake_case. 10950

Metricbeat

• Fix issue in kubernetes module preventing usage percentages to be properly calculated. 10946

Packetbeat

• Avoid reporting unknown MongoDB opcodes more than once. 10878

Winlogbeat

• Prevent Winlogbeat from dropping events with invalid XML. 11006

Affecting all Beats

• Fix stopping of modules started by kubernetes autodiscover. 10476

Auditbeat

• Enable System module config on Windows. 10237

Filebeat

• Fix bad bytes count in docker input when filtering by stream. 10211

• Add convert_timezone option to Logstash module to convert dates to UTC. 9756 9797

• Add convert_timezone option to Elasticsearch module to convert dates to UTC. 9756 9761

• Make elasticsearch/audit fileset be more lenient in parsing node name. 10035 10135

Journalbeat

• Fix fields.yml indentation of audit group which had the effect of creating an incomplete Elasticsearch index template. 10556

Metricbeat

• Fix issue with elasticsearch/node_stats metricset (x-pack) not indexing source_node field. 10639

Packetbeat

• Fixed a crash when using af_packet capture 10477

Functionbeat

• Ensure that functionbeat is logging at info level not debug. 10262

Filebeat

• Teach elasticsearch/audit fileset to parse out some more fields. 10134 10137

Journalbeat

• Migrate registry from previously incorrect path. 10486

Affecting all Beats

• Dissect syntax change, use * instead of ? when working with field reference. 8054

Filebeat

• Allow beats to blacklist certain part of the configuration while using Central Management. 9099

Metricbeat

• Allow beats to blacklist certain part of the configuration while using Central Management. 9099

Functionbeat

• The CLI will now log CloudFormation Stack events. 8912

• Correctly normalize Cloudformation resource name. 10087

Affecting all Beats

• Fix autodiscover configurations stopping when metadata is missing. 8851

• Refresh host metadata in add_host_metadata. 9359

• When collecting swap metrics for beats telemetry or system metricbeat module handle cases of free swap being bigger than total swap by assuming no swap is being used. 6271 9383

• Ignore non index fields in default_field for Elasticsearch. 9549

• Update Golang to 1.10.6. 9563

• Update Kibana index pattern attributes for objects that are disabled. 9644

• Enforce validation for the Central Management access token. 9621

• Fix registry handle leak on Windows (https://github.com/elastic/go-sysinfo/pull/33). 9920

• Gracefully handle TLS options when enrolling a Beat. 9129

• Allow to unenroll a Beat from the UI. 9452

• The backing off now implements jitter to better distribute the load. 10172

• Fix config appender registration. 9873

• Fix TLS certificate DoS vulnerability. 10304

Filebeat

• Fix improperly set config for CRI Flag in Docker Input 8899

• Just enabling the elasticsearch fileset and starting Filebeat no longer causes an error. 8891

• Fix macOS default log path for elasticsearch module based on homebrew paths. {pul}8939[8939]

• Support IPv6 addresses with zone id in IIS ingest pipeline. 9836 error log: 9869 access log: 10030

• Support haproxy log lines without captured headers. 9463 9958

Heartbeat

• Heartbeat now always downloads the entire body of HTTP endpoints, even if no checks against the body content are declared. This fixes an issue where timing metrics would be incorrect in scenarios where the body wasn’t used since the connection would be closed soon after the headers were sent, but before the entire body was. 8894

Metricbeat

• Add missing namespace field in http server metricset 7890

• Fix issue with not collecting Elasticsearch cross-cluster replication stats correctly. 9179

• The node.name field in the elasticsearch/node metricset now correctly reports the Elasticsarch node name. Previously this field was incorrectly reporting the node ID instead. 9209

• Fix panics in vsphere module when certain values where not returned by the API. 9784

• Fix pod UID metadata enrichment in Kubernetes module. 10081

Packetbeat

• Fix issue with process monitor associating traffic to the wrong process. 9151 9443

• Fix DHCPv4 dashboard that wouldn’t load in Kibana. 9850

Affecting all Beats

• Unify dashboard exporter tools. 9097

• Dissect will now flag event on parsing error. 8751

• Added the redirect_stderr option that allows panics to be logged to log files. 8430

• Add cache.ttl to add_host_metadata. 9359

• Add support for index lifecycle management (beta). 7963

• Always include Pod UID as part of Pod metadata. {pull]9517[9517]

• Release Jolokia autodiscover as GA. 9706

Auditbeat

• Add system module. 9546

Filebeat - Added detect_null_bytes selector to detect null bytes from a io.reader. 9210 - Added syslog_host variable to HAProxy module to allow syslog listener to bind to configured host. 9366 - Allow to force CRI format parsing for better performance 8424 - Add event.dataset to module events. 9457 - Add field log.source.address and log.file.path to replace source. 9435 - Add support for multi-core thread_id in postgresql module 9156 9482 - Added netflow input type that supports NetFlow v1, v5, v6, v7, v8, v9 and IPFIX. 9399

Journalbeat

• Add the ability to check against JSON HTTP bodies with conditions. 8667

• Add cursor_seek_fallback option. 9234

Metricbeat

• Collect custom cluster display_name in elasticsearch/cluster_stats metricset. 8445

• Test etcd module with etcd 3.3. 9068

• All elasticsearch metricsets now have module-level cluster.id and cluster.name fields. 8770 8771 9164 9165 9166 9168

• All elasticsearch node-level metricsets now have node.id and node.name fields. 9168 9209

• Add settings to disable docker and cgroup cpu metrics per core. 9187 9194 9589

• The elasticsearch/node metricset now reports the Elasticsearch cluster UUID. 8771

• Support GET requests in Jolokia module. 8566 9226

• Add freebsd support for the uptime metricset. 9413

• Add host.os.name field to add_host_metadata processor. 8948 9405

• Add field event.dataset which is {module}.{metricset). 9393

Filebeat - Deprecate field source. Will be replaced by log.source.address and log.file.path in 7.0. 9435

Metricbeat

• Deprecate field metricset.rtt. Replaced by event.duration which is in nano instead of micro seconds. 9393

Packetbeat

• Support new TLS version negotiation introduced in TLS 1.3. 8647.

Affecting all Beats

• Update Golang to 1.10.6. This fixes an issue in remote certificate validation CVE-2018-16875. 9563

Filebeat

• Fix saved objects in filebeat haproxy dashboard. 9417

• Fixed a memory leak when harvesters are closed. 7820

Filebeat

• Added support on Traefik for Common Log Format and Combined Log Format mixed which is the default Traefik format 8015 6111 8768.

Affecting all Beats

• Log events at the debug level when dropped by encoding problems. 9251

Filebeat

• Correctly parse December or Dec in the Syslog input. 9349

• Don’t generate incomplete configurations when logs collection is disabled by hints. 9305

• Stop runners disabled by hints after previously being started. 9305

• Fix installation of haproxy dashboard. 9307 9313

Affecting all Beats

• Propagate Sync error when running SafeFileRotate. 9069

Metricbeat

• Fix panic on docker healthcheck collection on dockers without healthchecks. 9171

• Fix issue preventing diskio metrics collection for idle disks. 9124 9125

Affecting all Beats - Fix windows binaries not having an enroll command. 9096 8836

Journalbeat - Fix journalbeat sometimes hanging if output is unavailable. 9106

Metricbeat - Fix race condition when enriching events with kubernetes metadata. 9055 9067

Journalbeat - Add minimal kibana dashboard. 9106

Affecting all Beats

• Fixed add_host_metadata not initializing correctly on Windows. 7715

• Fixed missing file unlock in spool file on Windows, so file can be reopened and locked. 7859

• Fix spool file opening/creation failing due to file locking on Windows. 7859

• Fix size of maximum mmaped read area in spool file on Windows. 7859

• Fix potential data loss on OS X in spool file by using fcntl with F_FULLFSYNC. 7859

• Improve fsync on linux, by assuming the kernel resets error flags of failed writes. 7859

• Remove unix-like permission checks on Windows, so files can be opened. 7849

• Replace index patterns in TSVB visualizations. 7929

• Deregister pipeline loader callback when inputsRunner is stopped. 7893[7893]

• Add backoff support to x-pack monitoring outputs. 7966

• Removed execute permissions systemd unit file. 7873

• Fix a race condition with the add_host_metadata and the event serialization. 8223 8653

• Enforce that data used by k8s or docker doesn’t use any reference. 8240

• Switch to different UUID lib due to to non-random generated UUIDs. 8485

• Fix race condition when publishing monitoring data. 8646

• Fix bug in loading dashboards from zip file. 8051

• Fix in-cluster kubernetes configuration on IPv6. 8754

• The export config subcommand should not display real value for field reference. 8769

• The setup command will not fail if no dashboard is available to import. 8977

• Fix central management configurations reload when a configuration is removed in Kibana. 9010

Auditbeat

• Fixed a crash in the file_integrity module under Linux. 7753

• Fixed the RPM by designating the config file as configuration data in the RPM spec. 8075

• Fixed a concurrent map write panic in the auditd module. 8158

• Fixed a data race in the file_integrity module. 8009

• Fixed a deadlock in the file_integrity module. 8027

Filebeat

• Fix date format in Mongodb Ingest pipeline. 7974

• Fixed a docker input error due to the offset update bug in partial log join.https://github.com/elastic/beats/pull/8177[8177]

• Update CRI format to support partial/full tags. 8265

• Fix some errors happening when stopping syslog input. 8347

• Fix RFC3339 timezone and nanoseconds parsing with the syslog input. 8346

• Mark the TCP and UDP input as GA. 8125

• Support multiline logs in logstash/log fileset of Filebeat. 8562

• Support different timestamp format in postgresql module. 9494 9650

Heartbeat

• Fixed bug where HTTP responses with larger bodies would incorrectly report connection errors. 8660

Metricbeat

• Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. 7789

• Fixed the RPM by designating the modules.d config files as configuration data in the RPM spec. 8075

• Fixed the location of the modules.d dir in Deb and RPM packages. 8104

• Add docker diskio stats on Windows. 6815 8126

• Fix incorrect type conversion of average response time in Haproxy dashboards 8404

• Added io disk read and write times to system module 8473 8508

• Avoid mapping issues in kubernetes module. 8487

• Recover metrics for old apache versions removed by mistake on #6450. 7871

• Fix dropwizard module parsing of metric names. 8365 8385

• Fix issue that would prevent kafka module to find a proper broker when port is not set 8613

• Fix range colors in multiple visualizations. 8633 8634

• Fix incorrect header parsing on http metricbeat module 8564 8585

• Fixed a panic when the kvm module cannot establish a connection to libvirtd. 7792.

Packetbeat

• Fixed a seccomp related error where the fcntl64 syscall was not permitted on 32-bit Linux and the sniffer failed to start. 7839

• Added missing cmdline and client_cmdline fields to index template. 8258

Affecting all Beats

• Added time-based log rotation. 8349

• Add backoff on error support to redis output. 7781

• Allow for cloud-id to specify a custom port. This makes cloud-id work in ECE contexts. 7887

• Add support to grow or shrink an existing spool file between restarts. 7859

• Make kubernetes autodiscover ignore events with empty container IDs 7971

• Implement CheckConfig in RunnerFactory to make autodiscover check configs 7961

• Add DNS processor with support for performing reverse lookups on IP addresses. 7770

• Support for Kafka 2.0.0 in kafka output 8399

• Add setting setup.kibana.space.id to support Kibana Spaces 7942

• Better tracking of number of open file descriptors. 7986

• Report number of open file handles on Windows. 8329

• Added the add_process_metadata processor to enrich events with process information. 6789

• Add Beats Central Management 8559

• Report configured queue type. 8091

• Enable host and cloud metadata processors by default. 8596

Filebeat

• Add tag "truncated" to "log.flags" if incoming line is longer than configured limit. 7991

• Add haproxy module. 8014

• Add tag "multiline" to "log.flags" if event consists of multiple lines. 7997

• Release docker input as GA. 8328

• Keep unparsed user agent information in user_agent.original. 7832

• Added default and TCP parsing formats to HAproxy module 8311 8637

• Add Suricata IDS/IDP/NSM module. 8153 8693

• Support for Kafka 2.0.0 8853

Heartbeat

• Heartbeat is marked as GA.

• Add automatic config file reloading. 8023

• Added autodiscovery support 8415

• Added support for extra TLS/x509 metadata. 7944

• Added stats and state metrics for number of monitors and endpoints started. 8621

• Add last monitor status to dashboard table. Further break out monitors in dashboard table by monitor.ip. 9022

Journalbeat

• Add journalbeat. 8703

Metricbeat

• Add replstatus metricset to MongoDB module 7604

• Add experimental socket summary metricset to system module 6782

• Move common kafka fields (broker, topic and partition.id) to the module level to facilitate events correlation 7767

• Add fields for memory fragmentation, memory allocator stats, copy on write, master-slave status, and active defragmentation to info metricset of Redis module. 7695

• Increase ignore_above for system.process.cmdline to 2048. 8100

• Add support to renamed fields planned for redis 5.0. 8167

• Allow TCP helper to support delimiters and graphite module to accept multiple metrics in a single payload. 8278

• Added 'died' PID state to process_system metricset on system module 8275

• Add metrics metricset to MongoDB module. 7611

• Added ccr metricset to Elasticsearch module. 8335

• Support for Kafka 2.0.0 8399

• Added support for query params in configuration 8286 8292

• Add container image for docker metricsets. 8214 8438

• Precalculate composed id fields for kafka dashboards. 8504

• Add support for full status page output for php-fpm module as a separate metricset called process. 8394

• Add Kafka dashboard. 8457

• Release Kafka module as GA. 8854

Packetbeat

• Added DHCP protocol support. 7647

Functionbeat

• Initial version of Functionbeat. 8678

Heartbeat

• watch.poll_file is now deprecated and superceded by automatic config file reloading.

Metricbeat

• Redis info replication.master_offset has been deprecated in favor of replication.master.offset.https://github.com/elastic/beats/pull/7695[7695]

• Redis info clients fields longest_output_list and biggest_input_buf have been renamed to max_output_buffer and max_input_buffer based on the names they will have in Redis 5.0, both fields will coexist during a time with the same value 8167.

• Move common kafka fields (broker, topic and partition.id) to the module level 7767.

Affecting all Beats

• Fix a race condition with the add_host_metadata and the event serialization. 8223 8653

• Fix race condition when publishing monitoring data. 8646

• Fix bug in loading dashboards from zip file. 8051

• The export config subcommand should not display real value for field reference. 8769

Filebeat

• Fix typo in Filebeat IIS Kibana visualization. 8604

Metricbeat

• Recover metrics for old Apache versions removed by mistake on #6450. 7871

• Avoid mapping issues in Kubernetes module. 8487

• Fixed a panic when the KVM module cannot establish a connection to libvirtd. 7792

Filebeat

• Fix some errors happening when stopping syslog input. 8347

• Fix RFC3339 timezone and nanoseconds parsing with the syslog input. 8346

Metricbeat

• Fix incorrect type conversion of average response time in Haproxy dashboards 8404

• Fix dropwizard module parsing of metric names. 8365 8385

Affecting all Beats

• Add backoff support to x-pack monitoring outputs. 7966

• Removed execute permissions systemd unit file. 7873

• Fix a race condition with the add_host_metadata and the event serialization. 8223

• Enforce that data used by k8s or docker doesn’t use any reference. 8240

• Implement CheckConfig in RunnerFactory to make autodiscover check configs 7961

• Make kubernetes autodiscover ignore events with empty container IDs 7971

Auditbeat

• Fixed a concurrent map write panic in the auditd module. 8158

• Fixed the RPM by designating the config file as configuration data in the RPM spec. 8075

Filebeat

• Fixed a docker input error due to the offset update bug in partial log join.https://github.com/elastic/beats/pull/8177[8177]

• Update CRI format to support partial/full tags. 8265

Metricbeat

• Fixed the location of the modules.d dir in Deb and RPM packages. 8104

• Fixed the RPM by designating the modules.d config files as configuration data in the RPM spec. 8075

• Fix golang.heap.gc.cpu_fraction type from long to float in Golang module. 7789

Packetbeat

• Added missing cmdline and client_cmdline fields to index template. 8258

Due to a packaging mistake, the modules.d configuration directory is installed in the wrong path in the Metricbeat DEB and RPM packages. This issue results in an empty list when you run metricbeat modules list and failures when you try to enable or disable modules. To work around this issue, run the following command:

This issue affects all new installations on DEB and RPM. Upgrades will run, but use old configurations defined in the modules.d directory from the previous installation.

The issue will be fixed in the 6.4.1 release.

Affecting all Beats

• Set default kafka version to 1.0.0 in kafka output. Older versions are still supported by configuring the version setting. Minimally supported version is 0.11 (older versions might work, but are untested). 7025

Heartbeat

• Rename http.response.status to http.response.status_code to align with ECS. 7274

• Remove type field as not needed. 7307

Metricbeat

• Fixed typo in values for state_container status.phase, from terminate to terminated. 6916

• RabbitMQ management plugin path is now configured at the module level instead of having to do it in each of the metricsets. New management_path_prefix option should be used now 7074

• RabbitMQ node metricset only collects metrics of the instance it connects to, node.collect: cluster can be used to collect all nodes as before. 6556 6971

• Change http/server metricset to put events by default under http.server and prefix config options with server.. 7100

• Disable dedotting in docker module configuration. This will change the out-of-the-box behaviour, but not the one of already configured instances. 7485

• Fix typo in etcd/self metricset fields from *.bandwithrate to *.bandwidthrate. 7456

• Changed the definition of the system.cpu.total.pct and system.cpu.total.norm.cou fields to exclude the IOWait time. 7691

Affecting all Beats

• Error out on invalid Autodiscover template conditions settings. 7200

• Allow to override the ignore_above option when defining new field with the type keyword. 7238

• Fix a panic on the Dissect processor when we have data remaining after the last delimiter. 7449

• When we fail to build a Kubernetes' indexer or matcher we produce a warning but we don’t add them to the execution. 7466

• Fix default value for logging.files.keepfiles. It was being set to 0 and now it’s set to the documented value of 7. 7494

• Retain compatibility with older Docker server versions. 7542

• Fix errors unpacking configs modified via CLI by ignoring -E key=value pairs with missing value. 7599

Auditbeat

• Allow auditbeat setup to run without requiring elevated privileges for the audit client. 7111

• Fix goroutine leak that occurred when the auditd module was stopped. 7163

Filebeat

• Fix a data race between stopping and starting of the harvesters. 6879

• Fix an issue when parsing ISO8601 dates with timezone definition 7367

• Fix Grok pattern of MongoDB module. 7568

• Fix registry duplicates and log resending on upgrade. 7634

Metricbeat

• Fix Windows service metricset when using a 32-bit binary on a 64-bit OS. 7294

• Do not report Metricbeat container host as hostname in Kubernetes deployment. 7199

• Ensure metadata updates don’t replace existing pod metrics. 7573

• Fix kubernetes pct fields reporting. 7677

• Add support for new kube_node_status_condition in Kubernetes state_node. 7699

Affecting all Beats

• Add dissect processor. 6925

• Add IP-addresses and MAC-addresses to add_host_metadata. 6878

• Added a seccomp (secure computing) filter on Linux that whitelists the necessary system calls used by each Beat. 5213

• Ship fields.yml as part of the binary 4834

• Added options to dev-tools/cmd/dashboards/export_dashboard.go: -indexPattern to include index-pattern in output, -quiet to be quiet. 7101

• Add Indexer indexing by pod uid. Enable pod uid metadata gathering in add_kubernetes_metadata. Extended Matcher log_path matching to support volume mounts 7072

• Add default_fields to Elasticsearch template when connecting to Elasticsearch >= 7.0. 7015

• Add support for loading a template.json file directly instead of using fields.yml. 7039

• Add support for keyword multifields in field.yml. 7131

• Add experimental Jolokia Discovery autodiscover provider. 7141

• Add owner object info to Kubernetes metadata. 7231

• Add Beat export dashboard command. 7239

• Add support for docker autodiscover to monitor containers on host network 6708

• Add ability to define input configuration as stringified JSON for autodiscover. 7372

• Add processor definition support for hints builder 7386

• Add support to disable html escaping in outputs. 7445

• Refactor error handing in schema.Apply(). 7335

• Add additional types to Kubernetes metadata 7457

• Add module state reporting for Beats Monitoring. 7075

• Release the rename processor as GA. 7656

• Add support for Openstack Nova in add_cloud_metadata processor. 7663

• Add support to set Beats services to automatic-delayed start on Windows. 8711

Auditbeat

• Added XXH64 hash option for file integrity checks. 7311

• Added the show auditd-rules and show auditd-status commands to show kernel rules and status. 7114

• Add Kubernetes specs for auditbeat file integrity monitoring 7642

Filebeat

• Add Kibana module with log fileset. 7052

• Support MySQL 5.7.19 by mysql/slowlog 6969

• Correctly join partial log lines when using docker input. 6967

• Add support for TLS with client authentication to the TCP input 7056

• Converted part of pipeline from treafik/access metricSet to dissect to improve efficiency. 7209

• Add GC fileset to the Elasticsearch module. 7305

• Add Audit log fileset to the Elasticsearch module. 7365

• Add Slow log fileset to the Elasticsearch module. 7473

• Add deprecation fileset to the Elasticsearch module. 7474

• Add convert_timezone option to Kafka module to convert dates to UTC. 7546 7578

• Add patterns for kafka 1.1 logs. 7608

• Move debug messages in tcp input source 7712

Metricbeat

• Add experimental Elasticsearch index metricset. 6881

• Add dashboards and visualizations for haproxy metrics. 6934

• Add Jolokia agent in proxy mode. 6475

• Add message rates to the RabbitMQ queue metricset 6442 6606

• Add exchanges metricset to the RabbitMQ module 6442 6607

• Add Elasticsearch index_summary metricset. 6918

• Add shard metricset to Elasticsearch module. 7006

• Add apiserver metricset to Kubernetes module. 7059

• Add maxmemory to redis info metricset. 7127

• Set guest as default user in RabbitMQ module. 7107

• Add postgresql statement metricset. 7048 7060

• Update state_container metricset to support latest kube-state-metrics version. 7216

• Add TLS support to MongoDB module. 7401

• Added Traefik module with health metricset. 7413

• Add Elasticsearch ml_job metricsets. 7196

• Add support for bearer token files to HTTP helper. 7527

• Add Elasticsearch index recovery metricset. 7225

• Add locks, global_locks, oplatencies and process fields to status metricset of MongoDB module. 7613

• Run Kafka integration tests on version 1.1.0 7616

• Release raid and socket metricset from system module as GA. 7658

• Release elasticsearch module and all its metricsets as beta. 7662

• Release munin and traefik module as beta. 7660

• Add envoyproxy module. 7569

• Release prometheus collector metricset as GA. 7660

• Add Elasticsearch cluster_stats metricset. 7638

• Added basepath setting for HTTP-based metricsets 7700

• Add couchdb module. 9406

Packetbeat

• The process monitor now reports the command-line for all processes, under Linux and Windows. 7135

• Updated the TLS protocol parser with new cipher suites added to TLS 1.3. 7455

• Flows are enriched with process information using the process monitor. 7507

• Added UDP support to process monitor. 7571

Metricbeat

• Kubernetes state_container cpu.limit.nanocores and cpu.request.nanocores have been deprecated in favor of cpu.*.cores. 6916

Affecting all Beats

• Allow index-pattern only setup when setup.dashboards.only_index=true. 7285

• Preserve the event when source matching fails in add_docker_metadata. 7133

• Negotiate Docker API version from our client instead of using a hardcoded one. 7165

• Fix duplicating dynamic_fields in template when overwriting the template. 7352

Auditbeat

• Fixed parsing of AppArmor audit messages. 6978

Filebeat

• Comply with PostgreSQL database name format 7198

• Optimize PostgreSQL ingest pipeline to use anchored regexp and merge multiple regexp into a single expression. 7269

• Keep different registry entry per container stream to avoid wrong offsets. 7281

• Fix offset field pointing at end of a line. 6514

• Commit registry writes to stable storage to avoid corrupt registry files. 6792

Metricbeat

• Fix field mapping for the system process CPU ticks fields. 7230

• Ensure canonical naming for JMX beans is disabled in Jolokia module. 7047

• Fix Jolokia attribute mapping when using wildcards and MBean names with multiple properties. 7321

Packetbeat

• Fix an out of bounds access in HTTP parser caused by malformed request. 6997

• Fix missing type for http.response.body field. 7169

Auditbeat

• Added caching of UID and GID values to auditd module. 6978

• Updated syscall tables for Linux 4.16. 6978

• Added better error messages for when the auditd module fails due to the Linux kernel not supporting auditing (CONFIG_AUDIT=n). 7012

Metricbeat

• Collect accumulated docker network metrics and mark old ones as deprecated. 7253

Affecting all Beats

• De dot keys of labels and annotations in kubernetes meta processors to prevent collisions. 6203

• Rename beat.cpu..time metrics to beat.cpu..time.ms. 6449

• Add host.name field to all events, to avoid mapping conflicts. This could be breaking Logstash configs if you rely on the host field being a string. 7051

Filebeat

• Add validation for Stdin, when Filebeat is configured with Stdin and any other inputs, Filebeat will now refuse to start. 6463

• Mark system.syslog.message and system.auth.message as text instead of keyword. 6589

Metricbeat

• De dot keys in kubernetes/event metricset to prevent collisions. 6203

• Add config option for windows/perfmon metricset to ignore non existent counters. 6432

• Refactor docker CPU calculations to be more consistent with docker stats. 6608

• Update logstash.node_stats metricset to write data under logstash.node.stats.*. 6714

Affecting all Beats

• Fix panic when Events containing a float32 value are normalized. 6129

• Fix setup.dashboards.always_kibana when using Kibana 5.6. 6090

• Fix for Kafka logger. 6430

• Remove double slashes in Windows service script. 6491

• Ensure Kubernetes labels/annotations don’t break mapping 6490

• Ensure that the dashboard zip files can’t contain files outside of the kibana directory. 6921

• Fix map overwrite panics by cloning shared structs before doing the update. 6947

• Fix delays on autodiscovery events handling caused by blocking runner stops. 7170

• Do not emit Kubernetes autodiscover events for Pods without IP address. 7235

• Fix self metrics when containerized 6641

Auditbeat

• Add hex decoding for the name field in audit path records. 6687

• Fixed a deadlock in the file_integrity module under Windows. 6864

• Fixed parsing of AppArmor audit messages. 6978

• Allow auditbeat setup to run without requiring elevated privileges for the audit client. 7111

• Fix goroutine leak that occurred when the auditd module was stopped. 7163

Filebeat

• Fix panic when log prospector configuration fails to load. 6800

• Fix memory leak in log prospector when files cannot be read. 6797

• Add raw JSON to message field when JSON parsing fails. 6516

• Commit registry writes to stable storage to avoid corrupt registry files. 6877

• Fix a parsing issue in the syslog input for RFC3339 timestamp and time with nanoseconds. 7046

• Fix an issue with an overflowing wait group when using the TCP input. 7202

Heartbeat

• Fix race due to updates of shared a map, that was not supposed to be shared between multiple go-routines. 6616

Metricbeat

• Fix the default configuration for Logstash to include the default port. 6279

• Fix dealing with new process status codes in Linux kernel 4.14+. 6306

• Add filtering option by exact device names in system.diskio. diskio.include_devices. 6085

• Add connections metricset to RabbitMQ module 6548

• Fix panic in http dependent modules when invalid config was used. 6205

• Fix system.filesystem.used.pct value to match what df reports. 5494

• Fix namespace disambiguation in Kubernetes state_* metricsets. 6281

• Fix Windows perfmon metricset so that it sends metrics when an error occurs. 6542

• Fix Kubernetes calculated fields store. https://github.com/elastic/beats/pull/6564{6564}

• Exclude bind mounts in fsstat and filesystem metricsets. 6819

• Don’t stop Metricbeat if aerospike server is down. 6874

• disk reads and write count metrics in RabbitMQ queue metricset made optional. 6876

• Add mapping for docker metrics per cpu. 6843

Winlogbeat

• Fixed a crash under Windows 2003 and XP when an event had less insert strings than required by its format string. 6247

• Fix config validation to allow event_logs.processors. [pull]6217[6217]

Affecting all Beats

• Update Golang 1.9.4 6326

• Add the ability to log to the Windows Event Log. 5913

• The node name can be discovered automatically by machine-id matching when beat deployed outside Kubernetes cluster. 6146

• Panics will be written to the logger before exiting. 6199

• Add builder support for autodiscover and annotations builder 6408

• Add plugin support for autodiscover builders, providers 6457

• Preserve runtime from container statuses in Kubernetes autodiscover 6456

• Experimental feature setup.template.append_fields added. 6024

• Add appender support to autodiscover 6469

• Add add_host_metadata processor 5968

• Retry configuration to load dashboards if Kibana is not reachable when the beat starts. 6560

• Add has_fields conditional to filter events based on the existence of all the given fields. 6285 6653

• Add support for spooling to disk to the beats event publishing pipeline. 6581

• Added logging of system info at Beat startup. 5946

• Do not log errors if X-Pack Monitoring is enabled but Elastisearch X-Pack is not. 6627

• Add rename processor. 6292

• Allow override of dynamic template match_mapping_type for fields with object_type. 6691

Filebeat

• Add IIS module to parse access log and error log. 6127

• Renaming of the prospector type to the input type and all prospectors are now moved to the input folder, to maintain backward compatibility type aliasing was used to map the old type to the new one. This change also affect YAML configuration. 6078

• Addition of the TCP input 6700

• Add option to convert the timestamps to UTC in the system module. 5647

• Add Logstash module support for main log and the slow log, support the plain text or structured JSON format 5481

• Add stream filtering when using docker prospector. 6057

• Add support for CRI logs format. 5630

• Add json.ignore_decoding_error config to not log json decoding erors. 6547

• Make registry file permission configurable. 6455

• Add MongoDB module. 6238

• Add Ingest pipeline loading to setup. 6814

• Add support of log_format combined to NGINX access logs. 6858

• Release config reloading feature as GA.

• Add support human friendly size for the UDP input. 6886

• Add Syslog input to ingest RFC3164 Events via TCP and UDP 6842

• Remove the undefined username option from the Redis input and clarify the documentation. 6662

Heartbeat

• Made the URL field of Heartbeat aggregateable. 6263

• Use match.Matcher for checking Heartbeat response bodies with regular expressions. 6539

Metricbeat

• Support apache status pages for versions older than 2.4.16. 6450

• Add support for huge pages on Linux. 6436

• Support to optionally 'de dot' keys in http/json metricset to prevent collisions. 5970

• Add graphite protocol metricbeat module. 4734

• Add http server metricset to support push metrics via http. 4770

• Make config object public for graphite and http server 4820

• Add system uptime metricset. 4848

• Add experimental queue metricset to RabbitMQ module. 4788

• Add additional php-fpm pool status kpis for Metricbeat module 5287

• Add etcd module. 4970

• Add ip address of docker containers to event. 5379

• Add ceph osd tree information to metricbeat 5498

• Add ceph osd_df to metricbeat 5606

• Add basic Logstash module. 5540

• Add dashboard for Windows service metricset. 5603

• Add pct calculated fields for Pod and container CPU and memory usages. 6158

• Add statefulset support to Kubernetes module. 6236

• Refactor prometheus endpoint parsing to look similar to upstream prometheus 6332

• Making the http/json metricset GA. 6471

• Add support for array in http/json metricset. 6480

• Making the jolokia/jmx module GA. 6143

• Making the MongoDB module GA. 6554

• Allow to disable labels dedot in Docker module, in favor of a safe way to keep dots. 6490

• Add experimental module to collect metrics from munin nodes. 6517

• Add support for wildcards and explicit metrics grouping in jolokia/jmx. 6462

• Set collector as default metricset in Prometheus module. 6636 6747

• Set mntr as default metricset in Zookeeper module. 6674

• Set default metricsets in vSphere module. 6676

• Set status as default metricset in Apache module. 6673

• Set namespace as default metricset in Aerospike module. 6669

• Set service as default metricset in Windows module. 6675

• Set all metricsets as default metricsets in uwsgi module. 6688

• Allow autodiscover to monitor unexposed ports 6727

• Mark kubernetes.event metricset as beta. 6715

• Set all metricsets as default metricsets in couchbase module. 6683

• Mark uwsgi module and metricset as beta. 6717

• Mark Golang module and metricsets as beta. 6711

• Mark system.raid metricset as beta. 6710

• Mark http.server metricset as beta. 6712

• Mark metricbeat logstash module and metricsets as beta. 6713

• Set all metricsets as default metricsets in Ceph module. 6676

• Set container, cpu, diskio, healthcheck, info, memory and network in docker module as default. 6718

• Set cpu, load, memory, network, process and process_summary as default metricsets in system module. 6689

• Set collector as default metricset in Dropwizard module. 6669

• Set info and keyspace as default metricsets in redis module. 6742

• Set connection as default metricset in rabbitmq module. 6743

• Set all metricsets as default metricsets in Elasticsearch module. 6755

• Set all metricsets as default metricsets in Etcd module. 6756

• Set server metricsets as default in Graphite module. 6757

• Set all metricsets as default metricsets in HAProxy module. 6758

• Set all metricsets as default metricsets in Kafka module. 6759

• Set all metricsets as default metricsets in postgresql module. 6761

• Set status metricsets as default in Kibana module. 6762

• Set all metricsets as default metricsets in Logstash module. 6763

• Set container, node, pod, system, volume as default in Kubernetes module. https://github.com/elastic/beats/pull/ 6764[6764]

• Set stats as default in memcached module. 6765

• Set all metricsets as default metricsets in Mongodb module. 6766

• Set pool as default metricset for php_fpm module. 6768

• Set status as default metricset for mysql module. https://github.com/elastic/beats/pull/ 6769[6769]

• Set stubstatus as default metricset for nginx module. 6770

• Added support for haproxy 1.7 and 1.8. 6793

• Add accumulated I/O stats to diskio in the line of docker stats. 6701

• Ignore virtual filesystem types by default in system module. 6819

• Release config reloading feature as GA. 6891

• Kubernetes deployment: Add ServiceAccount config to system metricbeat. 6824

• Kubernetes deployment: Add DNS Policy to system metricbeat. 6656

Packetbeat

• Add support for condition on bool type 5659 5954

• Fix high memory usage on HTTP body if body is not published. 6680

• Allow to capture the HTTP request or response bodies independently. 6784

• HTTP publishes an Error event for unmatched requests or responses. 6794

Winlogbeat

• Use bookmarks to persist the last published event. 6150

Affecting all Beats

• Fix conditions checking on autodiscover Docker labels. 6412

Affecting all Beats

• Avoid panic errors when processing nil Pod events in add_kubernetes_metadata. 6372

• Fix infinite failure on Kubernetes watch 6504

Metricbeat

• Fix Kubernetes overview dashboard views for non default time ranges. https://github.com/elastic/beats/issues/6395{6395}

Affecting all Beats

• Add logging when monitoring cannot connect to Elasticsearch. 6365

• Fix infinite loop when event unmarshal fails in Kubernetes pod watcher. 6353

Filebeat

• Fix a conversion issue for time related fields in the Logstash module for the slowlog fileset. 6317

Affecting all Beats

• The log format may differ due to logging library changes. 5901

• The default value for pipelining is reduced to 2 to avoid high memory in the Logstash beats input. 6250

Auditbeat

• Split the audit.kernel and audit.file metricsets into their own modules named auditd and file_integrity, respectively. This change requires existing users to update their config. 5422

• Renamed file_integrity module fields. 5423 5995

• Renamed auditd module fields. 5423 6080

Metricbeat

• Rename golang.heap.system.optained field to golang.heap.system.obtained. 5703

• De dot keys in jolokia/jmx metricset to prevent collisions. 5957

Auditbeat

• Fixed an issue where the proctitle value was being truncated. 6080

• Fixed an issue where values were incorrectly interpreted as hex data. 6080

• Fixed parsing of the key value when multiple keys are present. 6080

• Fix possible resource leak if file_integrity module is used with config reloading on Windows or Linux. 6198

Filebeat

• Fix variable name for convert_timezone in the system module. 5936

Metricbeat

• Fix error datastore '*' not found in Vsphere module. 4879

• Fix error NotAuthenticated in Vsphere module. 4673

• Fix mongodb session consistency mode to allow command execution on secondary nodes. 4689

• Fix kubernetes state_pod status.phase so that the active phase is returned instead of unknown. 5980

• Fix error collecting network_names in Vsphere module. 5962

• Fix process cgroup memory metrics for memsw, kmem, and kmem_tcp. 6033

• Fix kafka OffsetFetch request missing topic and partition parameters. 5880

Packetbeat

• Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572

Affecting all Beats

• Adding a local keystore to allow user to obfuscate password 5687

• Add autodiscover for kubernetes. 6055

• Add Beats metrics reporting to Xpack. 3422

• Update the command line library cobra and add support for zsh completion 5761

• Update to Golang 1.9.2

• Moved ip_port indexer for add_kubernetes_metadata to all beats. 5707

• ip_port indexer now index both IP and IP:port pairs. 5721

• Add the ability to write structured logs. 5901

• Use structured logging for the metrics that are periodically logged via the logging.metrics feature. 5915

• Improve Elasticsearch output metrics to count number of dropped and duplicate (if event ID is given) events. 5811

• Add the ability for the add_docker_metadata process to enrich based on process ID. 6100

• The add_docker_metadata and add_kubernetes_metadata processors are now GA, instead of Beta. 6105

• Update go-ucfg library to support top level key reference and cyclic key reference for the keystore 6098

Auditbeat

• Auditbeat is marked as GA, no longer Beta. 5432

• Add support for BLAKE2b hash algorithms to the file integrity module. 5926

• Add support for recursive file watches. 5575 5833

Filebeat

• Add Osquery module. 5971

• Add stream filtering when using docker prospector. 6057

Metricbeat

• Add ceph osd_df to metricbeat 5606

• Add field network_names of hosts and virtual machines. 5646

• Add experimental system/raid metricset. 5642

• Add a dashboard for the Nginx module. 5991

• Add experimental mongodb/collstats metricset. 5852

• Update the MySQL dashboard to use the Time Series Visual Builder. 5996

• Add experimental uwsgi module. 6006

• Docker and Kubernetes modules are now GA, instead of Beta. 6105

• Support haproxy stats gathering using http (additionally to tcp socket). 5819

• Support to optionally 'de dot' keys in http/json metricset to prevent collisions. 5957

Packetbeat

• Configure good defaults for add_kubernetes_metadata. 5707

Auditbeat

• Add an error check to the file integrity scanner to prevent a panic when there is an error reading file info via lstat. 6005

Filebeat

• Switch to docker prospector in sample manifests for Kubernetes deployment 5963

Auditbeat

• Changed audit.file.path to be a multi-field so that path is searchable. 5625

Metricbeat

• Rename heap_init field to heap.init in the Elasticsearch module. 5320

• Rename http.response.status_code field to http.response.code in the HTTP module. 5521

Affecting all Beats

• Remove ID() from Runner interface 5153

• Correctly send configured Host header to the remote server. 4842

• Change add_kubernetes_metadata to attempt detection of namespace. 5482

• Avoid double slash when join url and path 5517

• Fix console color output for Windows. 5611

• Fix logstash output debug message. https://github.com/elastic/beats/pull/5799{5799]

• Fix isolation of modules when merging local and global field settings. 5795

• Report ephemeral ID and uptime in monitoring events on all platforms 6501

Filebeat

• Add support for adding string tags 5395

• Fix race condition when limiting the number of harvesters running in parallel 5458

• Fix relative paths in the prospector definitions. 5443

• Fix recursive_globe.enabled option. 5443

Metricbeat

• Change field type of http header from nested to object 5258

• Fix the fetching of process information when some data is missing under MacOS X. 5337

• Change MySQL active connections visualization title to MySQL total connections. 4812

• Fix ProcState on Linux and FreeBSD when process names contain parentheses. 5775

• Fix incorrect Mem.Used calculation under linux. 5775

• Fix open_file_descriptor_count and max_file_descriptor_count lost in zookeeper module 5902

• Fix system process metricset for kernel processes. 5700

• Change kubernetes.node.cpu.allocatable.cores to float. 6130

Packetbeat

• Fix http status phrase parsing not allow spaces. 5312

• Fix http parse to allow to parse get request with space in the URI. 5495

• Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572

• Fix corruption when parsing repeated headers in an HTTP request or response. 6325

• Fix panic when parsing partial AMQP messages. 6384

• Fix out of bounds access to slice in MongoDB parser. 6256

• Fix sniffer hanging on exit under Linux. 6535

• Fix bounds check error in http parser causing a panic. 6750

Winlogbeat

• Fix the registry file. It was not correctly storing event log names, and upon restart it would begin reading at the start of each event log. 5813

Affecting all Beats

• Support dashboard loading without Elasticsearch 5653

• Changed the hashbang used in the beat helper script from /bin/bash to /usr/bin/env bash. 5051

• Changed beat helper script to use exec when running the beat. 5051

• Fix reloader error message to only print on actual error 5066

• Add support for enabling TLS renegotiation. 4386

• Add Azure VM support for add_cloud_metadata processor 5355

• Add output.file.permission config option. 4638

• Refactor add_kubernetes_metadata to support autodiscovery 5434

• Improve custom flag handling and CLI flags usage message. 5543

• Add number_of_routing_shards config set to 30 5570

• Set log level for kafka output. 5397

• Move TCP UDP start up into server.Start() 4903

• Update to Golang 1.9.2

Auditbeat

• Add support for SHA3 hash algorithms to the file integrity module. 5345

• Add dashboards for Linux audit framework events (overview, executions, sockets). 5516

Filebeat

• Add PostgreSQL module with slowlog support. 4763

• Add Kafka log module. 4885

• Add support for /var/log/containers/ log path in add_kubernetes_metadata processor. 4981

• Remove error log from runnerfactory as error is returned by API. 5085

• Add experimental Docker json-file prospector . 5402

• Add experimental Docker autodiscover functionality. 5245

• Add option to convert the timestamps to UTC in the system module. 5647

• Add Logstash module support for main log and the slow log, support the plain text or structured JSON format 5481

Metricbeat

• Add graphite protocol metricbeat module. 4734

• Add http server metricset to support push metrics via http. 4770

• Make config object public for graphite and http server 4820

• Add system uptime metricset. 4848

• Add experimental queue metricset to RabbitMQ module. 4788

• Add additional php-fpm pool status kpis for Metricbeat module 5287

• Add etcd module. 4970

• Add ip address of docker containers to event. 5379

• Add ceph osd tree information to Metricbeat 5498

• Add basic Logstash module. 5540

• Add dashboard for Windows service metricset. 5603

• Add experimental Docker autodiscover functionality. 5245

• Add Windows service metricset in the windows module. 5332

• Update gosigar to v0.6.0. 5775

Packetbeat

• Add support for decoding the TLS envelopes. 5476

• HTTP parses successfully on empty status phrase. 6176

• HTTP parser supports broken status line. 6631

Affecting all Beats

• Fix documentation links in README.md files. 5710

• Fix add_docker_metadata dropping some containers. 5788

Heartbeat

• Fix the "HTTP up status" visualization. 5564

Metricbeat

• Fix map overwrite in docker diskio module. 5582

• Fix connection leak in mongodb module. 5688

• Fix the include top N processes feature for cases where there are fewer processes than N. 5729

libbeat/docs/release-notes/6.0.0.asciidoc

Filebeat

• Fix machine learning jobs setup for dynamic modules. 5509

Packetbeat

• Fix missing length check in the PostgreSQL module. 5457

• Fix panic in ACK handler if event is dropped on blocked queue 5524

Filebeat

• Add Kubernetes manifests to deploy Filebeat. 5349

• Add container short ID matching to add_docker_metadata. 6172

Metricbeat

• Add Kubernetes manifests to deploy Metricbeat. 5349

Packetbeat

• Remove not-working runoptions.uid and runoptions.gid options in Packetbeat. 5261

Affecting all Beats

• Fix data race accessing watched containers. 5147

• Do not require template if index change and template disabled 5319

• Fix missing ACK in redis output. 5404

Filebeat

• Fix default paths for redis 4.0.1 logs on macOS 5173

• Fix Filebeat not starting if command line and modules configs are used together. 5376

• Fix double @timestamp field when JSON decoding was used. 5436

Metricbeat

• Use beat.name instead of beat.hostname in the Host Overview dashboard. 5340

• Fix the loading of 5.x dashboards. 5277

Metricbeat

• Auto-select a hostname (based on the host on which the Beat is running) in the Host Overview dashboard. 5340

Filebeat

• The filebeat.config_dir option is deprecated. Use filebeat.config.prospector options instead. 5321

Affecting all Beats

• Fix the /usr/bin/beatname script to accept -d "*" as a parameter. 5040

• Combine fields.yml properties when they are defined in different sources. 5075

• Keep Docker & Kubernetes pod metadata after container dies while they are needed by processors. 5084

• Fix fields.yml lookup when using export template with a custom path.config param. 5089

• Remove runner creation from every reload check 5141

• Fix add_kubernetes_metadata matcher registry lookup. 5159

Metricbeat

• Fix a memory allocation issue where more memory was allocated than needed in the windows-perfmon metricset. 5035

• Don’t start metricbeat if external modules config is wrong and reload is disabled 5053

• The MongoDB module now connects on each fetch, to avoid stopping the whole Metricbeat instance if MongoDB is not up when starting. 5120

• Fix kubernetes events module to be able to index time fields properly. 5093

• Fixed cmd_set and cmd_get being mixed in the Memcache module. 5189

Affecting all Beats

• Enable flush timeout by default. 5150

• Add @metadata.version to events send to Logstash. 5166

Auditbeat

• Changed the number of shards in the default configuration to 3. 5095

• Add support for receiving audit events using a multicast socket. 4850

Filebeat

• Changed the number of shards in the default configuration to 3. 5095

• Don’t start filebeat if external modules/prospectors config is wrong and reload is disabled 5053

• Add filebeat.registry_flush setting, to delay the registry updates. 5146

Heartbeat

• Changed the number of shards in the default configuration to 1. 5095

Packetbeat

• Changed the number of shards in the default configuration to 3. 5095

Winlogbeat

• Changed the number of shards in the default configuration to 3. 5095

Affecting all Beats

• The log directory (path.log) for Windows services is now set to C:\ProgramData\[beatname]\logs. 4764

• The _all field is disabled in Elasticsearch 6.0. This means that searching by individual words only work on text fields. 4901

• Fail if removed setting output.X.flush_interval is explicitly configured.

• Rename the /usr/bin/beatname.sh script (e.g. metricbeat.sh) to /usr/bin/beatname. 4933

• Beat does not start if elasticsearch index pattern was modified but not the template name and pattern. 4769

• Fail if removed setting output.X.flush_interval is explicitly configured. 4880

Affecting all Beats

• Register kubernetes field_format matcher and remove logger in Encode API 4888

• Fix go plugins not loaded when beat starts 4799

• Add support for initContainers in add_kubernetes_metadata processor. 4825

• Eliminate deprecated default mapping in 6.x 4864

• Fix pod name indexer to use both namespace, pod name to frame index key 4775

Filebeat

• Fix issue where the fileset.module could have the wrong value. 4761

Heartbeat

• Fix monitor.name being empty by default. 4852

• Fix wrong event timestamps. 4851

Metricbeat

• Added missing mongodb configuration file to the modules.d folder. 4870

• Fix wrong MySQL CRUD queries timelion visualization 4857

• Add new metrics to CPU metricset 4969

Packetbeat

• Update flow timestamp on each packet being received. 4895

Affecting all Beats

• Add setting to enable/disable the slow start in logstash output. 4972

• Update init scripts to use the test config subcommand instead of the deprecated -configtest flag. 4600

• Get by default the credentials for connecting to Kibana from the Elasticsearch output configuration. 4867

• Added cloud.id and cloud.auth settings, for simplifying using Beats with the Elastic Cloud. 4959

• Add lz4 compression support to kafka output. 4977

• Add newer kafka versions to kafka output. 4977

• Configure the index name when loading the dashboards and the index pattern. 4949

Metricbeat

• Add filesystem.ignore_types to system module for ignoring filesystem types. 4685

• Add support to exclude labels from kubernetes pod metadata. 4757

Affecting all Beats

• Rename kubernetes processor to add_kubernetes_metadata. 4473

• Rename .full.yml config files to .reference.yml. 4563

• The scripts/import_dashboards is removed from packages. Use the setup command instead. 4586

• Change format of the saved kibana dashboards to have a single JSON file for each dashboard 4413

• Rename configtest command to test config. 4590

• Remove setting queue_size and bulk_queue_size. 4650

• Remove setting dashboard.snapshot and dashboard.snapshot_url. They are no longer needed because the dashboards are included in the packages by default. 4675

• Beats can no longer be launched from Windows Explorer (GUI), command line is required. 4420

Auditbeat

• Changed file metricset config to make file.paths a list instead of a dictionary. 4796

Heartbeat

• Renamed the heartbeat RPM/DEB name to heartbeat-elastic. 4601

Metricbeat

• Change all system.cpu.*.pct metrics to be scaled by the number of CPU cores. This will make the CPU usage percentages from the system cpu metricset consistent with the system process metricset. The documentation for these metrics already stated that on multi-core systems the percentages could be greater than 100%. 4544

• Remove filters setting from metricbeat modules. 4699

• Added type field to filesystem metrics. 4717

Packetbeat

• Remove the already unsupported pf_ring sniffer option. 4608

Affecting all Beats

• Don’t stop with error loading the ES template if the ES output is not enabled. 4436

• Fix race condition in internal logging rotator. 4519

• Normalize all times to UTC to ensure proper index naming. 4569

• Fix issue with loading dashboards to ES 6.0 when .kibana index did not already exist. 4659

Auditbeat

• Fix file.max_file_size config option for the audit file metricset. 4796

Filebeat

• Fix issue where the fileset.module could have the wrong value. 4761

Metricbeat

• Fix issue affecting Windows services timing out at startup. 4491

• Fix incorrect docker.diskio.total metric calculation. 4507

• Vsphere module: used memory field corrected. 4461

Packetbeat

• Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing. 4442

Winlogbeat

• Removed validation of top-level config keys. This behavior was inconsistent with other Beats and caused maintainability issues. 4657

Affecting all Beats

• New cli subcommands interface. 4420

• Allow source path matching in add_docker_metadata processor. 4495

• Add support for analyzers and multifields in fields.yml. 4574

• Add support for JSON logging. 4523

• Add test output command, to test Elasticsearch and Logstash output settings. 4590

• Introduce configurable event queue settings: queue.mem.events, queue.mem.flush.min_events and queue.mem.flush.timeout. 4650

• Enable pipelining in Logstash output by default. 4650

• Added 'result' field to Elasticsearch QueryResult struct for compatibility with 6.x Index and Delete API responses. {issue]4661[4661]

• The sample dashboards are now included in the Beats packages. 4675

• Add pattern option to be used in the fields.yml to specify the pattern for a number field. 4731

Auditbeat

• Added file.hash_types config option for controlling the hash types. 4796

• Added the ability to specify byte unit suffixes to file.max_file_size. 4796

Filebeat

• Add experimental Redis module. 4441

• Nginx module: use the first not-private IP address as the remote_ip. 4417

• Load Ingest Node pipelines when the Elasticsearch connection is established, instead of only once at startup. 4479

• Add support for loading Xpack Machine Learning configurations from the modules, and added sample configurations for the Nginx module. 4506 4609

• Add udp prospector type. 4452

• Enabled Cgo which means libc is dynamically compiled. 4546

• Add Beta module config reloading mechanism 4566

• Remove spooler and publisher components and settings. 4644

Heartbeat

• Enabled Cgo which means libc is dynamically compiled. 4546

Metricbeat

• Add random startup delay to each metricset to avoid the thundering herd problem. 4010

• Add the ability to configure audit rules to the kernel module. 4482

• Add the ability to configure kernel’s audit failure mode. 4516

• Add experimental Aerospike module. 4560

• Vsphere module: collect custom fields from virtual machines. 4464

• Add test modules command, to test modules expected output. 4656

• Add processors setting to metricbeat modules. 4699

• Support npipe protocol (Windows) in Docker module. 4751

Winlogbeat

• Add the ability to use LevelRaw if Level isn’t populated in the event XML. 4257

Auditbeat

• Add file integrity metricset to the audit module. 4486

Filebeat

• Rename input_type field to prospector.type 4294

• The @metadata.type field, added by the Logstash output, is now hardcoded to doc and will be removed in future versions. 4331.

Affecting all Beats

• Fix importing the dashboards when the limit for max open files is too low. 4244

• Fix configuration documentation for kubernetes processor 4313

• Fix misspelling in add_locale configuration option for abbreviation.

Filebeat

• Fix race condition on harvester stopping with reloading enabled. 3779

• Fix recursive glob config parsing and resolution across restarts. 4269

• Allow string characters in user agent patch version (NGINX and Apache) 4415

• Fix grok pattern in filebeat module system/auth without hostname. 4224

Metricbeat

• Set correct format for percent fields in memory module. 4619

• Fix a debug statement that said a module wrapper had stopped when it hadn’t. 4264

• Use MemAvailable value from /proc/meminfo on Linux 3.14. 4316

• Fix panic when events were dropped by filters. 4327

• Add filtering to system filesystem metricset to remove relative mountpoints like those from Linux network namespaces. 4370

• Remove unnecessary print statement in schema apis. 4355

• Fix type of field haproxy.stat.check.health.last. 4407

Packetbeat - Enable memcache filtering only if a port is specified in the config file. 4335 - Enable memcache filtering only if a port is specified in the config file. 4335

Affecting all Beats

• Upgraded to Golang 1.8.3. 4401

• Added the possibility to set Elasticsearch mapping template settings from the Beat configuration file. 4284 4317

• Add a variable to the SysV init scripts to make it easier to change the user. 4340

• Add the option to write the generated Elasticsearch mapping template into a file. 4323

• Add instance_name in GCE add_cloud_metadata processor. 4414

• Add add_docker_metadata processor. 4352

• Add logging.files permissions option. 4295

Filebeat - Added ability to sort harvested files. 4374 - Add experimental Redis slow log prospector type. 4180

Metricbeat

• Add macOS implementation of the system diskio metricset. 4144

• Add process_summary metricset that records high level metrics about processes. 4231

• Add kube-state-metrics based metrics to kubernetes module 4253

• Add debug logging to Jolokia JMX metricset. 4341

• Add events metricset for kubernetes metricbeat module 4315

• Change Metricbeat default configuration file to be better optimized for most users. 4329

• Add experimental RabbitMQ module. 4394

• Add Kibana dashboard for the Kubernetes modules. 4138

Packetbeat

Winlogbeat

Affecting all Beats

• The @metadata.type field, added by the Logstash output, is deprecated, hardcoded to doc and will be removed in future versions. 4331.

Filebeat

• Deprecate input_type prospector config. Use type config option instead. 4294

• If the Elasticsearch output is not enabled, but setup.template options are present (like it’s the case in the default Metricbeat configuration), the Beat stops with an error: "Template loading requested but the Elasticsearch output is not configured/enabled". To avoid this error, disable the template loading explicitly setup.template.enabled: false.

Affecting all Beats

• Introduce beat version in the Elasticsearch index and mapping template 3527

• Usage of field _type is now ignored and hardcoded to doc. 3757

• Change vendor manager from glide to govendor. 3851

• Rename error field to error.message. 3987

• Change dashboards. config options to setup.dashboards.. 3921

• Change outputs.elasticsearch.template.* to `setup.template.* 4080

Filebeat

• Remove code to convert states from 1.x. 3767

• Remove deprecated config options force_close_files and close_older. 3768

• Change clean_removed behaviour to also remove states for files which cannot be found anymore under the same name. 3827

• Remove document_type config option. Use fields instead. 4204

• Move json_error under error.message and error.key. 4167

Packetbeat

• Remove deprecated geoip. 3766

• Replace waitstop command line argument by shutdown_timeout in configuration file. 3588

Winlogbeat

• Remove metrics endpoint. Replaced by http endpoint in libbeat (see #3717). 3901

Affecting all Beats

• Add _id, _type, _index and _score fields in the generated index pattern. 3282

Filebeat

• Fix the Mysql slowlog parsing of IP addresses. 4183

• Fix issue that new prospector was not reloaded on conflict 4128

Heartbeat

• Use IP type of elasticsearch for ip field. 3926

Metricbeat

• Support common.Time in mapstriface.toTime() 3812

• Fix MongoDB dbstats fields mapping. 4025

• Fixing prometheus collector to aggregate metrics based on metric family. 4075

• Fixing multiEventFetch error reporting when no events are returned 4153

Affecting all Beats

• Initialize a beats UUID from file on startup. 3615

• Add new add_locale processor to export the local timezone with an event. 3902

• Add http endpoint. 3717

• Updated to Go 1.8.1. 4033

• Add kubernetes processor 3888

• Add support for include_labels and include_annotations in kubernetes processor 4043

• Support new index_patterns field when loading templates for Elasticsearch >= 6.0 4056

• Adding goimports support to make check and fmt 4114

• Make kubernetes indexers/matchers pluggable 4151

• Abstracting pod interface in kubernetes plugin to enable easier vendoring 4152

Filebeat

• Restructure input.Event to be inline with outputs.Data 3823

• Add base for supporting prospector level processors 3853

• Add filebeat.config.path as replacement for config_dir. 4051

• Add a recursive_glob.enabled setting to expand ** in patterns. 3980

• Add Icinga module. 3904

• Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address.

Heartbeat

• Event format and field naming changes in Heartbeat and sample Dashboard. 4091

Metricbeat

• Add experimental metricset perfmon to Windows module. 3758

• Add memcached module with stats metricset. 3693

• Add the process.cmdline.cache.enabled config option to the System Process Metricset. 3891

• Add new MetricSet interfaces for developers (Closer, ReportingFetcher, and PushMetricSet). 3908

• Add kubelet module 3916

• Add dropwizard module 4022

• Adding query APIs for metricsets and modules from metricbeat registry 4102

• Fixing nil pointer on prometheus collector when http response is nil 4119

• Add http module with json metricset. 4092

• Add the option to the system module to include only the first top N processes by CPU and memory. 4127.

• Add experimental Vsphere module. 4028

• Add experimental Elasticsearch module. 3903

• Add experimental Kibana module. 3895

• Move elasticsearch metricset node_stats under node.stats namespace. 4142

• Make IP port indexer constructor public 4434

Packetbeat

• Add fields and fields_under_root to Packetbeat protocols configurations. 3518

• Add list style Packetbeat protocols configurations. This change supports specifying multiple configurations of the same protocol analyzer. 3518

Winlogbeat

Affecting all Beats

• Usage of field _type is deprecated. It should not be used in queries or dashboards. 3409

Packetbeat

• Deprecate dictionary style protocols configuration. 3518

Winlogbeat

Filebeat

• Prospector reloading only works properly with new files. 3546

Packetbeat

• Fix an out of bounds access in HTTP parser caused by malformed request. 6997

Affecting all Beats

• Fix a type issue when specifying certicate authority when using the import_dashboards command. 6678

Packetbeat

• Fix http status phrase parsing not allow spaces. 5312

• Fix http parse to allow to parse get request with space in the URI. 5495

• Fix mysql SQL parser to trim \r from Windows Server SELECT\r\n\t1. 5572

• Fix corruption when parsing repeated headers in an HTTP request or response. 6325

• Fix panic when parsing partial AMQP messages. 6384

• Fix out of bounds access to slice in MongoDB parser. 6256

• Fix sniffer hanging on exit under Linux. 6535

• Fix bounds check error in http parser causing a panic. 6750

• HTTP parses successfully on empty status phrase. 6176

• HTTP parser supports broken status line. 6631

Winlogbeat

• Fixed a crash under Windows 2003 and XP when an event had less insert strings than required by its format string. 6247

Affecting all Beats

• Fix duplicate batches of events in retry queue. 5520

Metricbeat

• Clarify meaning of percentages reported by system core metricset. 5565

• Fix map overwrite in docker diskio module. 5582

Affecting all Beats

• Fix race condition in internal logging rotator. 4519

Packetbeat

• Fix missing length check in the PostgreSQL module. 5457

Affecting all Beats

• Add support for enabling TLS renegotiation. 4386

• Add setting to enable/disable the slow start in logstash output. 5400

Affecting all Beats

• The _all.norms setting in the Elasticsearch template is no longer disabled. This increases the storage size with one byte per document, but allows for a better upgrade experience to 6.0. 4901

Filebeat

• Fix issue where the fileset.module could have the wrong value. 4761

Packetbeat

• Update flow timestamp on each packet being received. 4895

Metricbeat

• Fix a debug statement that said a module wrapper had stopped when it hadn’t. 4264

• Use MemAvailable value from /proc/meminfo on Linux 3.14. 4316

• Fix panic when events were dropped by filters. 4327

Affecting all Beats

• Add option to the import_dashboards script to load the dashboards via Kibana API. 4682

Filebeat

• Add support for loading Xpack Machine Learning configurations from the modules, and added sample configurations for the Nginx module. 4506 4609

• Add ability to parse nginx logs exposing the X-Forwarded-For header instead of the remote address. 4351

Metricbeat

• Add filesystem.ignore_types to system module for ignoring filesystem types. 4685

Affecting all Beats

• Loading more than one output is deprecated and will be removed in 6.0. 4907

Affecting all Beats

• Normalize all times to UTC to ensure proper index naming. 4569

Affecting all Beats

• Usage of field _type is now ignored and hardcoded to doc. 3757

Metricbeat - Change all system.cpu.*.pct metrics to be scaled by the number of CPU cores. This will make the CPU usage percentages from the system cpu metricset consistent with the system process metricset. The documentation for these metrics already stated that on multi-core systems the percentages could be greater than 100%. 4544

Affecting all Beats

• Fix console output. 4045

Filebeat

• Allow string characters in user agent patch version (NGINX and Apache) 4415

Metricbeat

• Fix type of field haproxy.stat.check.health.last. 4407

Packetbeat

• Fix packetbeat.interface options that contain underscores (e.g. with_vlans or bpf_filter). 4378

• Enabled /proc/net/tcp6 scanning and fixed ip v6 parsing. 4442

Filebeat

• Deprecate document_type prospector config option as _type is removed in elasticsearch 6.0. Use fields instead. 4225

Winlogbeat

• Deprecated metrics endpoint. It is superseded by a libbeat feature that can serve metrics on an HTTP endpoint. 4145

Affecting all Beats

• Removed empty sections from the template files, causing indexing errors for array objects. 4488

Metricbeat

• Fix issue affecting Windows services timing out at startup. 4491

• Add filtering to system filesystem metricset to remove relative mountpoints like those from Linux network namespaces. 4370

Packetbeat