Protobuf-deserialization is broken

[whotwagner]

Currently I receive the following error when I send a LogScheme-data tot he parser:

parser-1     | [2026-01-16 20:33:39,247] ERROR service.core: Component processing error:
parser-1     | [2026-01-16 20:33:39,247] ERROR service.core: Component processing error:
parser-1     | [2026-01-16 20:33:39,247] DEBUG parsers.json_parser.JsonParser.69aeb178a6ed5dcdbeb78df5be5fd7c7: Engine: Received 148 bytes from socket
parser-1     | [2026-01-16 20:33:39,247] DEBUG parsers.json_parser.JsonParser.69aeb178a6ed5dcdbeb78df5be5fd7c7: Engine: Calling processor...
parser-1     | [2026-01-16 20:33:39,247] DEBUG parsers.json_parser.JsonParser.69aeb178a6ed5dcdbeb78df5be5fd7c7: Engine: Processor returned: None
parser-1     | [2026-01-16 20:33:39,247] DEBUG parsers.json_parser.JsonParser.69aeb178a6ed5dcdbeb78df5be5fd7c7: Engine: Processor returned None, skipping send
parser-1     | [2026-01-16 20:33:59,869] DEBUG parsers.json_parser.JsonParser.69aeb178a6ed5dcdbeb78df5be5fd7c7: Engine: Received 148 bytes from socket
parser-1     | [2026-01-16 20:33:59,869] DEBUG parsers.json_parser.JsonParser.69aeb178a6ed5dcdbeb78df5be5fd7c7: Engine: Calling processor...
parser-1     | [2026-01-16 20:33:59,869] ERROR service.core: Component processing error:
parser-1     | [2026-01-16 20:33:59,869] ERROR service.core: Component processing error:
parser-1     | [2026-01-16 20:33:59,870] DEBUG parsers.json_parser.JsonParser.69aeb178a6ed5dcdbeb78df5be5fd7c7: Engine: Processor returned: None
parser-1     | [2026-01-16 20:33:59,871] DEBUG parsers.json_parser.JsonParser.69aeb178a6ed5dcdbeb78df5be5fd7c7: Engine: Processor returned None, skipping send

In line

DetectMateLibrary/src/detectmatelibrary/common/core.py

Line 26 in 022603e

input_.deserialize(data)

we don't catch any of the exceptions that are defined in https://github.com/ait-detectmate/DetectMateLibrary/blob/main/src/detectmatelibrary/schemas/_op.py
Thats why we don't get any error message when an error occurs in deserialize. I tried to cache an error and I receive "NotSupportedSchema". So I looked into the deserialisation-function and I was able to locate the source of the error at

DetectMateLibrary/src/detectmatelibrary/schemas/_op.py

Line 132 in 022603e

schema_class = __get_schema_class(schema_id)

schema_class = __get_schema_class(schema_id)fails because schema_id is not, as expected, b'1' but b'x1a'

The printed data looks as follows: b'\x1ai{"time": 2023-11-18 10:30:00","message": "pid=9699 uid=0 auid=4294967295 ses=4294967295","level": "INFO"}"\x11/var/log/some.log*\x03dev'

When I try to deserialize the exact line with the following code in python it can be serialized:

def from_proto(data):
    log = schemas_pb2.LogSchema()
    log.ParseFromString(data)
    return log

I believe that the code in _op.py and _class.py is faulty. Maybe we can also simplify that code. It is very hard to read and to follow.

[whotwagner]

I was able to fix it by changing the id in line

DetectMateLibrary/src/detectmatelibrary/schemas/_op.py

Line 16 in fdc5c0a

LOG_SCHEMA: SchemaID = SchemaID(b"1")

LOG_SCHEMA: SchemaID = SchemaID(b"1")

to

LOG_SCHEMA: SchemaID = SchemaID(b"1a")

Please note that this is not a real fix, because in my tests once that changed to b"10". So we have to come up with a better solution. But after that I got a parsing error in line:

DetectMateLibrary/src/detectmatelibrary/schemas/_op.py

Line 134 in fdc5c0a

schema.ParseFromString(message[1:])

schema.ParseFromString(message[1:])

I was able to fix that permanently by changing that code to:

schema.ParseFromString(message)

..because we don't want to corrupt the data we received by cutting the first byte.

In general I am still thinking of optimizing that code. Because we call that code on every single logline. This might also be a performance issue. The reason why we have that code is, because we want to dynamically detect what kind of data we have. But I am confident that we don't need to detect what kind of data it is. A parser knows that it is a parser and that it requires a LogSchema as input. And a detector knows that it is a detector.. aso.
I believe we can remove the part with the BaseClass, the _op.py and the _class.py in generall. We can initialize a variable that knows what schema we need and then we just have to call what_ever_schema.ParseFromString()

[thorinaboenke]

https://github.com/ait-detectmate/DetectMateLibrary/pull/26/changes just added a few additional validations for ids to _op.py, is that actually related?

[whotwagner]

https://github.com/ait-detectmate/DetectMateLibrary/pull/26/changes just added a few additional validations for ids to _op.py, is that actually related?

no, when I tested this we didn't switch to str yet.

[ipmach]

Solved in #56 .