How to pass PcapPackets into Kafka queue

Question

With the below code to pass PcapPackets to a queue, is it possible to pass this into Kafka queue so that Kafka consumer can pull PcapPackets as such from Kafka producer?

StringBuilder errbuf = new StringBuilder();  
Pcap pcap = Pcap.openOffline("tests/test-afs.pcap", errbuf);  

PcapPacketHandler<Queue<PcapPacket>> handler = new PcapPacketHandler<Queue<PcapPacket>>() {  
  public void nextPacket(PcapPacket packet, Queue<PcapPacket> queue) {  
    PcapPacket permanent = new PcapPacket(packet);  

    queue.offer(packet);  
  }  
}  

Queue<PcapPacket> queue = new ArrayBlockingQueue<PcapPacket>();  

pcap.loop(10, handler, queue);  

System.out.println("we have " + queue.size() + " packets in our queue");  

pcap.close();

score 0 · Answer 1 · edited May 23 '17 at 10:33

0

Kafka supports storing an arbitrary binary data as messages. In your case you just need to provide a PcapPacket class binary serializer (and deserializer for reading).

See Kafka: writing custom serializer for an example.

edited May 23 '17 at 10:33

Community

1
1

answered Oct 14 '14 at 17:29

Denis Makarenko

2,853
15
29

score 0 · Answer 2 · answered May 10 '18 at 21:03

Though I am late to the party, I share my tool: Pcap Processor (GitHub URL) here if anyone with similar requirements finds it useful. I have developed a tool in Python for my research to read raw pcap files, to process them and to feed them to my stream processor. Since I tried various stream protocols, I implemented all of them in this tool. Currently supported sinks:

CSV file
Apache Kafka (encoded into JSON string)
HTTP REST (JSON)
gRPC
Console (just print to the terminal)

For example, to read input.pcap and to send it to a Kafka topic, you need to adjust the bootstrap endpoint and topic name in kafka_sink.py. Then, executing the following command from parent directory will read the file and send packets to Kafka queue.

python3 -m pcap_processor --sink kafka input.pcap

For more details and installation instructions, please check the GitHub readme and feel free to open GitHub issues if you encounter any problems.

How to pass PcapPackets into Kafka queue

2 Answers2