4

is there has any way to pass some field when parsing a truncated log in Kaitai Struct? Because if it read a field (type specify to a enum) but value not in there, it will raise a NullPointer Exception.
So I want ask if any way to achieve that just like default: pass attribute in python library Construct

Here is my ksy file:

meta:
  id: btsnoop
  endian: be
seq:
  - id: header
    type: header
  - id: packets
    type: packet
    repeat: eos
types:
  header:
    seq:
      - id: iden
        size: 8
      - id: version
        type: u4
      - id: datalink_type
        type: u4
        enum: linktype
  packet:
    seq:
      - id: ori_len
        type: u4
      - id: include_len
        type: u4
      - id: pkt_flags
        type: u4
      - id: cumu_drop
        type: u4
      - id: timestamp
        type: s8
      - id: data
        size: include_len
        type: frame
  frame:
    seq:
      - id: pkt_type
        type: u1
        enum: pkttype
      - id: cmd
        type: cmd
        if: pkt_type == pkttype::cmd_pkt
      - id: acl
        type: acl
        if: pkt_type == pkttype::acl_pkt
      - id: evt
        type: evt
        if: pkt_type == pkttype::evt_pkt  
  cmd:
    seq:
      - id: opcode
        type: u2le
      - id: params_len
        type: u1
      - id: params
        size: params_len
  acl:
    seq:
      - id: handle
        type: u2le
  evt:
    seq:
      - id: status
        type: u1
        enum: status
      - id: total_length
        type: u1
      - id: params
        size-eos: true
enums:  <-- I need to list all possible option in every enum?
  linktype:
    0x03E9: unencapsulated_hci
    0x03EA: hci_uart
    0x03EB: hci_bscp
    0x03EC: hci_serial
  pkttype:
    1: cmd_pkt
    2: acl_pkt
    4: evt_pkt
  status:
    0x0D: complete_D
    0x0E: complete_E
    0xFF: vendor_specific

Thanks for reply :)

Vince
  • 14,470
  • 7
  • 39
  • 84
Kaniel Venson
  • 181
  • 2
  • 12

1 Answers1

3

There are still two questions you're facing here :)

Parsing partial / truncated / damaged data

The main problem here is that normally Kaitai Struct compiles .ksy into a code that does the actual parsing in class constructor. That means if a problem arises, boom, you've got no object at all. In most use cases, it is desired behavior, as it actually allows you to be sure that the object is fully initialized. The problem is usually an EOFException, when format wants to read next primitive, but there's no data in the stream left, or, in some more complicated cases, something else.

However, there are some use-cases as you've mentioned, where having "best effort" parsing would be helpful - i.e. you're ok with having half-filled object. Another popular use case for that is the visualizer: it's helpful to show "best effort" there too, as it's better to show user half-parsed result visualized (to aid in locating at error) rather than no result at all (and leave the user with the guesswork).

There's a simple solution for that in Kaitai Struct - you can compile your class with --debug option. This way you'll get a class that has object creation and parsing separated, parsing would be just another method of an object (void _read()). However, this means that you'll have to call parsing method manually. For example, if your original code was:

Btssnoop b = Btssnoop.fromFile("/path/to/file.bin");
System.out.println(b.packets.size());

after you've compiled it with --debug, you'll have to do extra step:

Btssnoop b = Btssnoop.fromFile("/path/to/file.bin");
b._read();
System.out.println(b.packets.size());

and then you can wrap it up in a try/catch block and actually continue processing even after getting IOException:

Btssnoop b = Btssnoop.fromFile("/path/to/file.bin");
try {
    b._read();
} catch (IOException e) {
    System.out.println("warning: truncated packets");
}
System.out.println(b.packets.size());

There are a few catches, though:

  • --debug was not yet available for Java target, as of release v0.3; actually, it's not even in public git repository right now, I hope I'll push it soon though.
  • --debug also does a few extra things, like writing down positions of every attribute, which imposes pretty harsh performance / memory penalty. Tell me if you'll need a switch to compile "separate constructor/parsing" functionality without the rest of --debug functionality - I can think of additional switch to enable just that.
  • If you need to do continuous parsing incoming packets as they arrive, probably it's a bad idea to store them all in memory and re-parse them all on every update. We're considering event-based parsing model for that one, please tell me if you'd be interested in that one.

Missing enum values and NPE

Current Java implementation translates enums reading into something like

this.pet1 = Animal.byId(_io.readU4le());

where Animal.byId is translated into:

private static final Map<Long, Animal> byId = new HashMap<Long, Animal>(3);
static {
    for (Animal e : Animal.values())
        byId.put(e.id(), e);
}
public static Animal byId(long id) { return byId.get(id); }

Java Map's get returns null by contract, when no value was found in the map. You should be able to compare that null with something (i.e. other enum value) and get proper true or false. Can you show me where exactly you have NPE problem, i.e. your code, generated code and stack trace?

GreyCat
  • 16,622
  • 18
  • 74
  • 112