v2 - Breaking changes

Overview

OpenSearch has several breaking changes, so you must verify your application compatibility using this link:

Breaking changesOpenSearch documentation

Remove mapping type

This is the major breaking change and it is not specific to OpenSearch as it was alreay planned by ElasticSearch before the fork

Removal of mapping types | Elasticsearch Guide [7.17] | ElasticElastic

So you must be sure that your applications are not using anymore the "type" parameters.

Here are some solutions regarding products often use with elastic solutions and how to configure them to work with OpenSearch 2.x

Fluentbit

ElasticsearchFluent Bit: Official Manual

If the client is Fluentbit, the easiest solution is to set the parameter Suppress_Type_Name to On.

It is also possible to change the output plugin to the opensearch native one which is part of Fluentbit since version 1.9.

OpenSearchFluent Bit: Official Manual

The following article may prove useful for getting started with Fluentbit and OpenSearch:

https://opensearch.org/blog/technical/2022/03/getting-started-with-fluent-bit-and-opensearch/opensearch.org

Fluentd

If the client is Fluentd it's more tricky. There is also a suppress_type_name but the plugin is using this parameter only if detect an elastic version>=7.

So we need to add to more parameters:

• verify_es_version_at_startup to false to not let the plugin detect the version

• default_elasticsearch_version to '7'

GitHub - uken/fluent-plugin-elasticsearchGitHub

Here are for exemple the change to be done on the spec of the output plugin we're using in Kubernetes

Before

 elasticsearch:
    flatten_hashes: true
    host: ostest.es.cegedim.cloud
    include_tag_key: true
    log_es_400_reason: true
    logstash_format: true
    logstash_prefix: myit-app-prod-frontend
    password:
      valueFrom:
        secretKeyRef:
          key: myit-app-prod_password
          name: it-cloud-eb.es.cegedim.cloud
    port: 443
    prefer_oj_serializer: true
    reconnect_on_error: true
    request_timeout: 30s
    scheme: https
    suppress_type_name: true
    user: myit-app-prod

After

  elasticsearch:
    default_elasticsearch_version: "7"
    flatten_hashes: true
    host: ostest.es.cegedim.cloud
    include_tag_key: true
    log_es_400_reason: true
    logstash_format: true
    logstash_prefix: myit-app-prod-frontend
    password:
      valueFrom:
        secretKeyRef:
          key: myit-app-prod_password
          name: it-cloud-eb.es.cegedim.cloud
    port: 443
    prefer_oj_serializer: true
    reconnect_on_error: true
    request_timeout: 30s
    scheme: https
    suppress_type_name: true
    user: myit-app-prod
    verify_es_version_at_startup: false

There is also an output plugin for OpenSearch.

opensearchFluentd

The OpenSearch Plugin is not yet available in Rancher Logging System