4

While you can create a topic via Java or Java-based languages (see here), there does not seem to be a clean way to do this without using Java. As a result, pure-language client APIs (like kafka-node, a pure JavaScript client) can't directly create topics. Instead, we have two options:

1) Use a hack like sending a metadata request to a topic -- if auto.create.topics.enable is set to true, then you can create a topic -- but only with the default configuration, no control over partitions, etc.

2) Write a wrapper around a Java-based client just for topic creation. The easiest way to do this is to exec the script bin/kafka-topics.sh with command line arguments, which is ugly, to say the least.

Is there a better way to do this, though? There's a pure-JavaScript client for Zookeeper, node-zookeeper-client, what happens if I manipulate broker / partition info directly in Zookeeper?

Any other thoughts?

Community
  • 1
  • 1
David Griffin
  • 13,677
  • 5
  • 47
  • 65
  • There is another client, but with the same limitations for now http://docs.confluent.io/2.0.1/kafka-rest/docs/index.html – Nautilus Mar 13 '16 at 09:08
  • 1
    There is work being done in the Apache Kafka project on [KIP-4: Command line and centralized administrative operations](https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations) which will expose certain admin functions such as creating topics. Once KIP-4 is finished you should soon see better options for non-Java languages. As said by nautilus, Confluent's Kafka REST Proxy -- which is a popular tool for non-Java languages to interact with Kafka -- does not yet support creating Kafka topics; KIP-4 may help here, too. – miguno Mar 14 '16 at 12:52

1 Answers1

0

You can now use REST Proxy API v3 to create Kafka topics with http requests for non-Java languages.

According to the Confluent REST Proxy API Reference the creation of a topic is possible with the REST Proxy API v3 that is currently available as a preview feature.

"The API v3 can be used for evaluation and non-production testing purposes or to provide feedback to Confluent."

An example of a topic creation request is presented below and documented here:

POST /v3/clusters/cluster-1/topics HTTP/1.1
Host: kafkaproxy.example.com
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json

{
  "data": {
    "attributes": {
      "topic_name": "topic-1",
      "partitions_count": 2,
      "replication_factor": 3,
      "configs": [
        {
          "name": "cleanup.policy",
          "value": "compact"
        }
      ]
    }
  }
}

Using curl:

curl -X POST -H "Content-Type: application/vnd.api+json" -H "Accept: application/vnd.api+json" \
          --data '{"data":{"attributes": {"topic_name": "topic-1", "partitions_count": 2, "replication_factor": 1, "configs": [{"name": "cleanup.policy","value": "compact"}]}}}' \
          "http://localhost:8082/v3/clusters/<cluster-id>/topics"

where the cluster-id can be identified using

curl -X GET -H "Accept: application/vnd.api+json" localhost:8082/v3/clusters
Michael Heil
  • 16,250
  • 3
  • 42
  • 77
  • 1
    It is no longer a preview feature, and the new link to topic creation is https://docs.confluent.io/platform/current/kafka-rest/api.html#post--clusters-cluster_id-topics – Fogus May 09 '23 at 19:19