Affecting all Beats

• Update to Golang 1.12.1. 11330

• Update to Golang 1.12.4. 11782

• Update to ECS 1.0.1. 12284 12317

• Default of output.kafka.metadata.full is set to false by now. This reduced the amount of metadata to be queried from a kafka cluster. 12738

• Fixed a crash under Windows when fetching processes information. 12833

Auditbeat

• Auditd module: Normalized value of event.category field from user-login to authentication. 11432

• Auditd module: Unset auditd.session and user.audit.id fields are removed from audit events. 11431 11815

• Socket dataset: Exclude localhost by default 11993

Filebeat

• Add read_buffer configuration option. 11739

• convert_timezone option is removed and locale is always added to the event so timezone is used when parsing the timestamp, this behaviour can be overriden with processors. 12410

Heartbeat

• Removed the add_host_metadata and add_cloud_metadata processors from the default config. These don’t fit well with ECS for Heartbeat and were rarely used.

Journalbeat

Metricbeat

• Add new option OpMultiplyBuckets to scale histogram buckets to avoid decimal points in final events 10994

• system/raid metricset now uses /sys/block instead of /proc/mdstat for data. 11613

• kubernetes.container.cpu.limit.cores and kubernetes.container.cpu.requests.cores are now floats. 11975

Packetbeat

• Add support for mongodb opcode 2013 (OP_MSG). 6191 8594

• NFSv4: Always use opname ILLEGAL when failed to match request to a valid nfs operation. 11503

Winlogbeat

Functionbeat

Affecting all Beats

• Fix typo in TLS renegotiation configuration and setting the option correctly 10871, 12354

• Ensure all beat commands respect configured settings. 10721

• Add missing fields and test cases for libbeat add_kubernetes_metadata processor. 11133, 11134

• decode_json_field: process objects and arrays only 11312

• decode_json_field: do not process arrays when flag not set. 11318

• Report faulting file when config reload fails. 11304

• Fix a typo in libbeat/outputs/transport/client.go by updating c.conn.LocalAddr() to c.conn.RemoteAddr(). 11242

• Management configuration backup file will now have a timestamps in their name. 11034

• [CM] Parse enrollment_token response correctly 11648

• Not hiding error in case of http failure using elastic fetcher 11604

• Escape BOM on JsonReader before trying to decode line 11661

• Fix matching of string arrays in contains condition. 11691

• Replace wmi queries with win32 api calls as they were consuming CPU resources 3249 and 11840

• Fix a race condition with the Kafka pipeline client, it is possible that Close() get called before Connect() . 11945

• Fix queue.spool.write.flush.events config type. 12080

• Fixed a memory leak when using the add_process_metadata processor under Windows. 12100

• Fix of docker json parser for missing "log" jsonkey in docker container’s log 11464

• Fixed Beat ID being reported by GET / API. 12180

• Fixed setting bulk max size in kafka output. 12254

• Add host.os.codename to fields.yml. 12261

• Fix @timestamp being duplicated in events if @timestamp is set in a processor (or by any code utilizing PutValue() on a beat.Event).

• Fix leak in script processor when using Javascript functions in a processor chain. 12600

• Add additional nil pointer checks to Docker client code to deal with vSphere Integrated Containers 12628

• Fixed json.add_error_key property setting for delivering error messages from beat events 11298

• Fix Central Management enroll under Windows 12797 12799

Auditbeat

• Process dataset: Fixed a memory leak under Windows. 12100

• Login dataset: Fix re-read of utmp files. 12028

• Package dataset: Fixed a crash inside librpm after Auditbeat has been running for a while. 12147 12168

• Fix formatting of config files on macOS and Windows. 12148

• Fix direction of incoming IPv6 sockets. 12248

• Package dataset: Close librpm handle. 12215

• Package dataset: Auto-detect package directories. 12289

• Package dataset: Improve dpkg parsing. 12325

• System module: Start system module without host ID. 12373

• Host dataset: Fix reboot detection logic. 12591

• Add syscalls used by librpm for the system/package dataset to the default Auditbeat seccomp policy. 12578 12617

• Process dataset: Do not show non-root warning on Windows. 12740

Filebeat

• Add support for Cisco syslog format used by their switch. 10760

• Cover empty request data, url and version in Apache2 modulehttps://github.com/elastic/beats/pull/10730[10730]

• Fix registry entries not being cleaned due to race conditions. 10747

• Improve detection of file deletion on Windows. 10747

• Add missing Kubernetes metadata fields to Filebeat CoreDNS module, and fix a documentation error. 11591

• Reduce memory usage if long lines are truncated to fit max_bytes limit. The line buffer is copied into a smaller buffer now. This allows the runtime to release unused memory earlier. 11524

• Fix memory leak in Filebeat pipeline acker. 12063

• Fix goroutine leak caused on initialization failures of log input. 12125

• Fix goroutine leak on non-explicit finalization of log input. 12164

• Skipping unparsable log entries from docker json reader 12268

• Parse timezone in PostgreSQL logs as part of the timestamp 12338

• Load correct pipelines when system module is configured in modules.d. 12340

• Fix timezone offset parsing in system/syslog. 12529

• When TLS is configured for the TCP input and a certificate_authorities is configured we now default to required for the client_authentication. 12584

• Apply max_message_size to incoming message buffer. 11966

• Syslog input will now omit the process object from events if it is empty. 12700

Heartbeat

• Fix NPEs / resource leaks when executing config checks. 11165

• Fix duplicated IPs on mode: all monitors. 12458

Journalbeat

• Use backoff when no new events are found. 11861

• Iterate over journal correctly, so no duplicate entries are sent. 12716

• Preserve host name when reading from remote journal. 12714

Metricbeat

• Change diskio metrics retrieval method (only for Windows) from wmi query to DeviceIOControl function using the IOCTL_DISK_PERFORMANCE control code 11635

• Call GetMetricData api per region instead of per instance. 11820 11882

• Update documentation with cloudwatch:ListMetrics permission. 11987

• Check permissions in system socket metricset based on capabilities. 12039

• Get process information from sockets owned by current user when system socket metricset is run without privileges. 12039

• Avoid generating hints-based configuration with empty hosts when no exposed port is suitable for the hosts hint. 8264 12086

• Fixed a socket leak in the postgresql module under Windows when SSL is disabled on the server. 11393

• Change some field type from scaled_float to long in aws module. 11982

• Fixed RabbitMQ queue metricset gathering when consumer_utilisation is set empty at the metrics source 12089

• Fix direction of incoming IPv6 sockets. 12248

• Refactored Windows perfmon metricset: replaced method to retrieve counter paths with PdhExpandWildCardPathW, separated code by responsibility, removed unused functions 12212

• Validate that kibana/status metricset cannot be used when xpack is enabled. 12264

• Ignore prometheus metrics when their values are NaN or Inf. 12084 10849

• In the kibana/stats metricset, only log error (don’t also index it) if xpack is enabled. 12265

• Fix an issue listing all processes when run under Windows as a non-privileged user. 12301 12475

• The elasticsearch/index_summary metricset gracefully handles an empty Elasticsearch cluster when xpack.enabled: true is set. 12489 12487

• When TLS is configured for the http metricset and a certificate_authorities is configured we now default to required for the client_authentication. 12584

• Reuse connections in PostgreSQL metricsets. 12504 12603

• PdhExpandWildCardPathW will not expand counter paths in 32 bit windows systems, workaround will use a different function.https://github.com/elastic/beats/issues/12590[12590]12622

• In the elasticsearch/node_stats metricset, if xpack is enabled, make parsing of ES node load average optional as ES on Windows doesn’t report load average. 12866

• Ramdisk is not filtered out when collecting disk performance counters in diskio metricset 12814 12829

Packetbeat

• Prevent duplicate packet loss error messages in HTTP events. 10709

• Fixed a memory leak when using process monitoring under Windows. 12100

• Improved debug logging efficiency in PGQSL module. 12150

• Limit memory usage of Redis replication sessions. 12657

• Fix parsing the extended RCODE in the DNS parser. 12805

Winlogbeat

Functionbeat

• Fix function name reference for Kinesis streams in CloudFormation templates 11646

Affecting all Beats

• Decouple Debug logging from fail_on_error logic for rename, copy, truncate processors 12451

• Add an option to append to existing logs rather than always rotate on start. 11953

• Add network condition to processors for matching IP addresses against CIDRs. 10743

• Add if/then/else support to processors. 10744

• Add community_id processor for computing network flow hashes. 10745

• Add output test to kafka output 10834

• Gracefully shut down on SIGHUP 10704

• New processor: copy_fields. 11303

• Add error.message to events when fail_on_error is set in rename and copy_fields processors. 11303

• New processor: truncate_fields. 11297

• Allow a beat to ship monitoring data directly to an Elasticsearch monitoring cluster. 9260

• Updated go-seccomp-bpf library to v1.1.0 which updates syscall lists for Linux v5.0. NNNN

• Add add_observer_metadata processor. 11394

• Add decode_csv_fields processor. 11753

• Add convert processor for converting data types of fields. 8124 11686

• New extract_array processor. 11761

• Add number of goroutines to reported metrics. 12135

• Add proxy_disable output flag to explicitly ignore proxy environment variables. 11713 12243

• Processor add_cloud_metadata adds fields cloud.account.id and cloud.image.id for AWS EC2. 12307

• Add configurable bulk_flush_frequency in kafka output. 12254

• Add decode_base64_field processor for decoding base64 field. 11914

• Add support for reading the network.iana_number field by default to the community_id processor. 12701

• Add aws overview dashboard. 11007 12175

• Add decompress_gzip_field processor. 12733

• Add timestamp processor for parsing time fields. 12699

Auditbeat

• Auditd module: Add event.outcome and event.type for ECS. 11432

• Process: Add file hash of process executable. 11722

• Socket: Add network.transport and network.community_id. 12231

• Host: Fill top-level host fields. 12259

Filebeat

• Add more info to message logged when a duplicated symlink file is found 10845

• Add option to configure docker input with paths 10687

• Add Netflow module to enrich flow events with geoip data. 10877

• Set event.category: network_traffic for Suricata. 10882

• Allow custom default settings with autodiscover (for example, use of CRI paths for logs). 12193

• Allow to disable hints based autodiscover default behavior (fetching all logs). 12193

• Change Suricata module pipeline to handle destination.domain being set if a reverse DNS processor is used. 10510

• Add the network.community_id flow identifier to field to the IPTables, Suricata, and Zeek modules. 11005

• New Filebeat coredns module to ingest coredns logs. It supports both native coredns deployment and coredns deployment in kubernetes. 11200

• New module for Cisco ASA logs. 9200 11171

• Added support for Cisco ASA fields to the netflow input. 11201

• Configurable line terminator. 11015

• Add Filebeat envoyproxy module. 11700

• Add apache2(httpd) log path (/var/log/httpd) to make apache2 module work out of the box on Redhat-family OSes. 11887 11888

• Add support to new MongoDB additional diagnostic information 11952

• New module panw for Palo Alto Networks PAN-OS logs. 11999

• Add RabbitMQ module. 12032

• Add new container input. 12162

• Add timeouts on communication with docker daemon. 12310

• container and docker inputs now support reading of labels and env vars written by docker JSON file logging driver. 8358

• Add specific date processor to convert timezones so same pipeline can be used when convert_timezone is enabled or disabled. 12253

• Add MSSQL module 12079

• Add ISO8601 date parsing support for system module. 12568 12579

• Update Kubernetes deployment manifest to use container input. 12632

• Use correct OS path separator in add_kubernetes_metadata to support Windows nodes. 9205

• Add support for client addresses with port in Apache error logs 12695

• Add google-pubsub input type for consuming messages from a Google Cloud Pub/Sub topic subscription. 12746

• Add module for ingesting Cisco IOS logs over syslog. 12748

• Add module for ingesting Google Cloud VPC flow logs. 12747

• Add netflow dashboards based on Logstash netflow. 12857

Heartbeat

• Enable add_observer_metadata processor in default config. 11394

Journalbeat

Metricbeat

• Add AWS SQS metricset. 10684 10053

• Add AWS s3_request metricset. 10949 10055

• Add s3_daily_storage metricset. 10940 10055

• Add coredns metricbeat module. 10585

• Add SSL support for Metricbeat HTTP server. 11482 11457

• The elasticsearch.index metricset (with xpack.enabled: true) now collects refresh.external_total_time_in_millis fields from Elasticsearch. 11616

• Allow module configurations to have variants 9118

• Add timeseries.instance field calculation. 10293

• Added new disk states and raid level to the system/raid metricset. 11613

• Added path_name and start_name to service metricset on windows module 8364 11877

• Add check on object name in the counter path if the instance name is missing 6528 11878

• Add AWS cloudwatch metricset. 11798 11734

• Add regions in aws module config to specify target regions for querying cloudwatch metrics. 11932 11956

• Keep etcd followers members from reporting leader metricset events 12004

• Add overview dashboard to Consul module 10665

• New fields were added in the mysql/status metricset. 12227

• Add Kubernetes metricset proxy. 12312

• Add Kubernetes proxy dashboard to Kubernetes module 12734

• Always report Pod UID in the pod metricset. 12345

• Add Vsphere Virtual Machine operating system to os field in Vsphere virtualmachine module. 12391

• Add validation for elasticsearch and kibana modules' metricsets when xpack.enabled is set to true. 12386

• Add CockroachDB module. 12467

• Add support for metricbeat modules based on existing modules (a.k.a. light modules) 12270 12465

• Add a system/entropy metricset 12450

• Add kubernetes metricset controllermanager 12409

• Add Kubernetes controller manager dashboard to Kubernetes module 12744

• Allow redis URL format in redis hosts config. 12408

• Add tags into ec2 metricset. 1226312263 12372

• Add kubernetes metricset scheduler 12521

• Add Kubernetes scheduler dashboard to Kubernetes module 12749

• Add beat module. 12181 12615

• Collect tags for cloudwatch metricset in aws module. 1226312263 12480

• Add AWS RDS metricset. 11620 10054

• Add Oracle Module 11890

• Add Oracle Tablespaces Dashboard 12736

• Collect client provided name for rabbitmq connection. 12851 12852

Packetbeat

Functionbeat

• New options to configure roles and VPC. 11779

• Export automation templates used to create functions. 11923

• Configurable Amazon endpoint. 12369

Winlogbeat

• Add support for reading from .evtx files. 4450

Affecting all Beats

Filebeat

• docker input is deprecated in favour container. 12162

• postgresql.log.timestamp field is deprecated in favour of @timestamp. 12338

Heartbeat

Journalbeat

Metricbeat

Packetbeat

Winlogbeat

Functionbeat

Journalbeat