Add more parameters to opensearch output plugin

[discostur]

Add Extended OpenSearch Output Plugin Support

Summary

This PR significantly extends the OpenSearch output plugin CRD to support critical production features that were previously missing from the fluent-operator. Most notably, it adds support for log_os_400_reason, which is essential for debugging 400 errors from OpenSearch without enabling full debug logging.

Motivation

The current OpenSearch output plugin implementation only supported basic configuration options (host, port, credentials, SSL verification). This limited functionality made it difficult to:

• Debug 400 errors from OpenSearch (missing log_os_400_reason)
• Configure SSL/TLS properly for production environments
• Tune performance and reliability settings
• Handle connection failures gracefully
• Use advanced OpenSearch features like pipelines, templates, and custom write operations

Changes

Key Features Added

Debugging & Error Handling:

• logOs400Reason - Log 400 error reasons without debug level (critical for troubleshooting)
• reconnectOnError - Automatically reconnect on connection errors
• ignoreExceptions - Specify exception classes to ignore
• exceptionBackup - Backup chunks when exceptions occur

SSL/TLS Configuration:

• caFile, clientCert, clientKey, clientKeyPassword - Mutual TLS support
• sslVersion, sslMinVersion, sslMaxVersion - Fine-grained TLS version control

Connection Management:

• requestTimeout - HTTP request timeout configuration
• reloadConnections, reloadAfter, reloadOnFailure - Connection pooling control

Performance Tuning:

• compressionLevel - Enable gzip compression
• httpBackend - Choose HTTP backend (excon/typhoeus)
• preferOjSerializer - Use faster JSON serialization
• bulkMessageRequestThreshold - Control bulk request splitting

Record & Index Management:

• includeTagKey, tagKey - Include Fluentd tags in records
• idKey - Specify document ID field
• removeKeys, removeKeysOnUpdate - Remove sensitive fields
• writeOperation - Support create/update/upsert operations
• emitErrorForMissingId - Error handling for missing IDs
• pipeline - OpenSearch ingest pipeline support

Template & Version Management:

• templateOverwrite, maxRetryPuttingTemplate - Template management
• verifyOsVersionAtStartup, maxRetryGetOsVersion - Version detection
• defaultOpensearchVersion - Fallback version configuration
• applicationName, indexDatePattern - Rollover index support
• useLegacyTemplate - Support for both legacy and composable templates

Advanced Options:

• suppressTypeName - Handle OpenSearch 2.x type deprecation
• contentType - Configure Content-Type header
• customHeaders - Add custom HTTP headers
• snifferClassName, selectorClassName - Advanced node discovery
• suppressDocWrap - Control document wrapping
• utcIndex - UTC vs local time for indices

Modified Files

• apis/fluentd/v1alpha1/plugins/output/opensearch.go - Added 67 new configuration fields
• apis/fluentd/v1alpha1/plugins/output/types.go - Implemented serialization for all new fields
• config/crd/bases/fluentd.fluent.io_clusteroutputs.yaml - Generated CRD with new fields
• config/crd/bases/fluentd.fluent.io_outputs.yaml - Generated CRD with new fields
• charts/fluent-operator/charts/fluentd-crds/crds/*.yaml - Updated Helm chart CRDs
• manifests/setup/*.yaml - Updated manifest files
• docs/plugins/fluentd/output/opensearch.md - Updated documentation with all new parameters

Testing

• ✅ All existing OpenSearch tests pass (Test_Cfg2OpenSearch, Test_ClusterCfgOutput2OpenSearch, Test_MixedCfgs2OpenSearch)
• ✅ Code generation successful (make generate)
• ✅ CRD manifests generated successfully (make manifests)
• ✅ No breaking changes to existing configurations

Documentation

Updated comprehensive documentation including:

• Complete parameter reference with all new fields
• Production-ready example configuration
• Development/debug configuration example
• Migration guide from basic to extended configuration
• Troubleshooting section with common issues
• SSL/TLS configuration examples

Backward Compatibility

✅ Fully backward compatible - All new fields are optional with sensible defaults that match the plugin's behavior. Existing configurations will continue to work unchanged.

Usage Example

apiVersion: fluentd.fluent.io/v1alpha1
kind: ClusterOutput
metadata:
  name: opensearch-production
spec:
  outputs:
    - opensearch:
        hosts: "opensearch-node1:9200,opensearch-node2:9200"
        scheme: https
        
        # Authentication
        user:
          valueFrom:
            secretKeyRef:
              name: opensearch-creds
              key: username
        password:
          valueFrom:
            secretKeyRef:
              name: opensearch-creds
              key: password
        
        # CRITICAL: Enable 400 error debugging
        logOs400Reason: true
        
        # Production reliability
        reconnectOnError: true
        requestTimeout: 30s
        reloadConnections: true
        compressionLevel: best_speed
        
        # SSL/TLS
        sslVerify: true
        sslMinVersion: TLSv1_2
        sslMaxVersion: TLSv1_3
        caFile: /etc/ssl/opensearch/ca.crt

References

• fluent-plugin-opensearch - Official plugin documentation
• Resolves missing functionality compared to the upstream plugin

[discostur]

@marcofranssen thanks for the hint ;)

Refactor to use generic params.InsertPairs helper function

• Extended ValueType interface to support *int32 pointer type
• Replaced manual fmt.Sprint() calls with params.InsertPairs() in all OpenSearch helper functions
• Refactored kafka2Plugin to use params.InsertPairs()

[cw-Guo]

Hi @discostur , can you please sign off the commits? see details here

[discostur]

@cw-Guo done

[marcofranssen]

Suggested change

	ps.InsertPairs(key, strconv.FormatInt(int64(*v), 10))
	ps.InsertPairs(key, strconv.FormatInt(int32(*v), 10))

Not sure, looks like all we use int64, but not sure if that is correct either.

[discostur]

The conversion to int64/uint64 in these lines looks correct to me - it's required by strconv.FormatInt() and strconv.FormatUint().

The underlying inconsistency is in the type definitions across the project:

• Input ports: mostly *int32
• Output ports: mostly *uint32
• Exception: monitor_agent.Port uses *int64
• Other numeric fields: mixed *int16, *int32, *int64

The InsertPairs function must support all these types generically, hence the different cases. Standardizing types across the project could be a good idea for the future ;)

[kristiangronas]

Sorry if this is the wrong place to ask, but would it be easy to add template_file, @type opensearch_data_stream, data_stream_name and data_stream_template_name in this PR?

[discostur]

@kristiangronas @type opensearch_data_stream is a separat output plugin so this should go into a separate PR / issue.

[marcofranssen]

@discostur could you rebase the PR and resovle the conflicts?

git checkout master
git pull
git checkout your-branch
git rebase master
git push --force-with-lease

[discostur]

@marcofranssen done

[discostur]

synced with main

[marcofranssen]

Basic tests are failing. Can you fix those?

[discostur]

@marcofranssen done

[marcofranssen]

LGTM 🚀