Add new setting to control splitting log4j audit messages to prevent audit_trace_indices/audit_trace_resolved_indices from making messages too big

[parislarkins]

Add new setting plugins.security.audit.config.log4j.maximum_index_characters_per_message, when non-zero split audit log messages audit_trace_indices and audit_trace_resolved_indices into multiple messages that are under the maximum limit.

Description

Enhancement/New feature?

For clusters with large numbers of indices (e.g. ~11,000 in the case I first observed this issue) with the plugins.security.audit.config.resolve_indices OpenSearch setting and appender.rolling_audit.layout.maxMessageLength=0 log4j setting enabled (to disable log message truncation), the audit_trace_indices and audit_trace_resolved_indices fields in audit logs that relate to all indices can be extremely long. For example, 11,000 indexes with 100 character long names requires 1,100,000 characters to just to represent the index names alone (plus even more characters for the surrounding quotes and commas between each of them).

These messages can get so large as to cause problems for downstream parts of your logging pipeline (for example, the default Apache Kafka maximum message is 1mb). In these cases, it is usually recommended to split large messages into smaller ones, as they are able to be handled more efficiently than giant messages.

Old behaviour
Log4j audit messages log an unlimited number of index names in the audit_trace_indices and audit_trace_resolved_indices fields, leading to giant log messages for clusters with lots of indexes.

New behaviour
A simple example: If we set the new setting plugins.security.audit.config.log4j.maximum_index_characters_per_message to18, this original message:

{
  "audit_trace_indices": [
    "index*"
  ],
  "audit_trace_resolved_indices": [
    "index1",
    "index2",
    "index3",
    "index4",
    "index5",
    "index6",
    "index7",
    "index8"
  ],
  "audit_category": "AUTHENTICATED"
}

Would then be split into the following 3 messages:

{
  "audit_trace_indices": [
    "index*"
  ]
  "audit_trace_resolved_indices": [
    "index2",
    "index3"
  ],
  "audit_category": "AUTHENTICATED"
}

{
  "audit_trace_resolved_indices": [
    "index4",
    "index5",
    "index6"
  ],
  "audit_category": "AUTHENTICATED"
}

{
  "audit_trace_resolved_indices": [
    "index7",
    "index8"
  ],
  "audit_category": "AUTHENTICATED"
}

These 3 split messages contain the exact same information as the source message, so no information is lost (although it is more effort to re-construct the original event).

Issues Resolved

#5976

Testing

Unit tests added to AuditMessageTest class.

I have been running my fork of the plugin using this setting on live clusters successfully to reduce the maximum size of audit log messages so they can fit in the 1mb maximum Apache Kafka message size in my logging pipeline.

Check List

• New functionality includes testing
• New functionality has been documented
• New Roles/Permissions have a corresponding security dashboards plugin PR
• API changes companion pull request created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[codecov]

Codecov Report

❌ Patch coverage is 93.47826% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 73.83%. Comparing base (161df85) to head (e41eea9).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
...pensearch/security/auditlog/impl/AuditMessage.java	95.12%	2 Missing ⚠️
...g/opensearch/security/auditlog/sink/Log4JSink.java	66.66%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5977      +/-   ##
==========================================
- Coverage   73.87%   73.83%   -0.04%     
==========================================
  Files         439      439              
  Lines       27086    27132      +46     
  Branches     4018     4024       +6     
==========================================
+ Hits        20009    20034      +25     
- Misses       5171     5190      +19     
- Partials     1906     1908       +2     

Files with missing lines	Coverage Δ
.../opensearch/security/OpenSearchSecurityPlugin.java	85.10% <100.00%> (+0.03%)	⬆️
...g/opensearch/security/support/ConfigConstants.java	77.27% <ø> (ø)
...g/opensearch/security/auditlog/sink/Log4JSink.java	66.66% <66.66%> (+6.66%)	⬆️
...pensearch/security/auditlog/impl/AuditMessage.java	81.71% <95.12%> (+2.54%)	⬆️

... and 10 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.