Packages
brod
3.2.0
4.5.7
4.5.6
4.5.5
4.5.4
4.5.3
4.5.2
4.5.1
4.5.0
4.4.7
4.4.6
4.4.5
4.4.4
4.4.3
4.4.2
4.4.1
4.4.0
4.3.3
4.3.2
4.3.1
4.3.0
4.2.0
4.1.1
4.1.0
4.0.0
3.19.1
3.19.0
3.18.0
3.17.1
3.17.0
3.16.5
3.16.4
3.16.3
3.16.2
3.16.1
3.16.0
3.15.6
3.15.5
3.15.4
3.15.3
3.15.1
3.15.0
3.14.0
3.13.0
3.12.0
3.11.0
3.10.0
3.9.5
3.9.3
3.9.2
3.9.1
3.9.0
3.8.1
3.8.0
3.7.11
3.7.10
3.7.9
3.7.8
3.7.7
3.7.6
3.7.5
3.7.4
3.7.3
3.7.2
3.7.1
3.7.0
3.6.2
3.6.1
3.6.0
3.5.2
3.5.1
3.5.0
3.4.0
3.3.5
3.3.4
3.3.3
3.3.2
3.3.1
3.3.0
3.2.0
3.0.0
2.5.0
2.4.1
2.4.0
2.3.7
2.3.6
2.3.5
2.3.4
2.3.3
2.3.1
2.2.16
2.2.15
2.2.14
2.2.12
2.2.11
2.2.10
2.2.9
2.2.8
2.2.7
2.2.6
2.2.5
2.2.4
2.2.3
2.2.2
2.2.1
2.2.0
2.1.12
2.1.11
2.1.10
2.1.8
2.1.7
2.1.4
2.1.2
2.0.0
Apache Kafka Erlang client library
Current section
113 Versions
Jump to
Current section
113 Versions
Compare versions
22
files changed
+780
additions
-145
deletions
| @@ -1,6 +1,6 @@ | |
| 1 1 | PROJECT = brod |
| 2 2 | PROJECT_DESCRIPTION = Kafka client library in Erlang |
| 3 | - PROJECT_VERSION = 3.0.0 |
| 3 | + PROJECT_VERSION = 3.2.0 |
| 4 4 | |
| 5 5 | DEPS = supervisor3 kafka_protocol |
| 6 6 | TEST_DEPS = docopt jsone meck proper |
| @@ -10,7 +10,7 @@ ERLC_OPTS = -Werror +warn_unused_vars +warn_shadow_vars +warn_unused_import +war | |
| 10 10 | TEST_ERLC_OPTS = -Werror +warn_unused_vars +warn_shadow_vars +warn_unused_import +warn_obsolete_guard +debug_info |
| 11 11 | |
| 12 12 | dep_supervisor3_commit = 1.1.5 |
| 13 | - dep_kafka_protocol_commit = 1.0.0 |
| 13 | + dep_kafka_protocol_commit = 1.1.0 |
| 14 14 | dep_docopt = git https://github.com/zmstone/docopt-erl.git 0.1.3 |
| 15 15 | |
| 16 16 | ESCRIPT_FILE = scripts/brod |
| @@ -57,4 +57,3 @@ vsn-check: | |
| 57 57 | |
| 58 58 | hex-publish: distclean |
| 59 59 | $(verbose) rebar3 hex publish |
| 60 | - |
| @@ -17,11 +17,11 @@ Why "brod"? [http://en.wikipedia.org/wiki/Max_Brod](http://en.wikipedia.org/wiki | |
| 17 17 | * Simple consumer: The poller, has a configurable "prefetch count" - it will continue sending fetch requests as long as total number of unprocessed messages (not message-sets) is less than "prefetch count" |
| 18 18 | * Group subscriber: Support for consumer groups with options to have Kafka as offset storage or a custom one |
| 19 19 | * Topic subscriber: Subscribe on messages from all or selected topic partitions without using consumer groups |
| 20 | + * Pick latest supported version when sending requests to kafka. |
| 20 21 | |
| 21 22 | # Missing features |
| 22 23 | |
| 23 24 | * lz4 compression & decompression |
| 24 | - * new 0.10 on-wire message format |
| 25 25 | * new 0.10.1.0 create/delete topic api |
| 26 26 | |
| 27 27 | # Building and testing |
| @@ -29,6 +29,23 @@ Why "brod"? [http://en.wikipedia.org/wiki/Max_Brod](http://en.wikipedia.org/wiki | |
| 29 29 | make |
| 30 30 | make test-env t # requires docker-composer in place |
| 31 31 | |
| 32 | + # Working With Kafka 0.9.x or Earlier |
| 33 | + |
| 34 | + Make sure `{query_api_versions, false}` exists in client config. |
| 35 | + This is because `ApiVersionRequest` was introduced in kafka 0.10, |
| 36 | + sending such request to older version brokers will cause connection failure. |
| 37 | + |
| 38 | + e.g. in sys.config: |
| 39 | + |
| 40 | + ``` |
| 41 | + [{brod, |
| 42 | + [ { clients |
| 43 | + , [ { brod_client_1 %% registered name |
| 44 | + , [ { endpoints, [{"localhost", 9092}]} |
| 45 | + , { query_api_versions, false} %% <---------- here |
| 46 | + ]}]}]}] |
| 47 | + ``` |
| 48 | + |
| 32 49 | # Quick Demo |
| 33 50 | |
| 34 51 | Assuming kafka is running at `localhost:9092` |
| @@ -212,6 +229,13 @@ brod:produce(_Client = brod_client_1, | |
| 212 229 | ]). |
| 213 230 | ``` |
| 214 231 | |
| 232 | + ## Produce message create time. |
| 233 | + |
| 234 | + ```erlang |
| 235 | + brod:produce(Client, Topic, Partition, _Key = <<>>, |
| 236 | + _Value = [{Ts1, Key1, Value1}, {Ts2, Key2, Value2}]). |
| 237 | + ``` |
| 238 | + |
| 215 239 | ## Handle acks from kafka |
| 216 240 | |
| 217 241 | Unless brod:produce_sync was called, callers of brod:produce should |
| @@ -393,3 +417,31 @@ Start an Erlang shell with brod started | |
| 393 417 | ./_rel/brod/bin/brod-i |
| 394 418 | ``` |
| 395 419 | |
| 420 | + ## brod-cli examples: |
| 421 | + |
| 422 | + ### Fetch and print metadata |
| 423 | + ``` |
| 424 | + ./scripts/brod meta -b localhost |
| 425 | + ``` |
| 426 | + |
| 427 | + ### Produce a message |
| 428 | + ``` |
| 429 | + ./scripts/brod send -b localhost -t test-topic -p 0 -k "key" -v "value" |
| 430 | + |
| 431 | + ``` |
| 432 | + |
| 433 | + ### Fetch a message |
| 434 | + |
| 435 | + ``` |
| 436 | + ./scripts/brod fetch -b localhost -t test-topic -p 0 --fmt 'io:format("offset=~p, ts=~p, key=~s, value=~s\n", [Offset, Ts, Key, Value])' |
| 437 | + ``` |
| 438 | + |
| 439 | + Bound variables to be used in `--fmt` expression: |
| 440 | + |
| 441 | + - `Offset`: Message offset |
| 442 | + - `Key`: Kafka key |
| 443 | + - `Value`: Kafka Value |
| 444 | + - `CRC`: Message CRC |
| 445 | + - `TsType`: Timestamp type either `create` or `append` |
| 446 | + - `Ts`: Timestamp, `-1` as no value |
| 447 | + |
| @@ -1,9 +1,9 @@ | |
| 1 1 | {<<"name">>,<<"brod">>}. |
| 2 | - {<<"version">>,<<"3.0.0">>}. |
| 2 | + {<<"version">>,<<"3.2.0">>}. |
| 3 3 | {<<"requirements">>, |
| 4 4 | #{<<"kafka_protocol">> => #{<<"app">> => <<"kafka_protocol">>, |
| 5 5 | <<"optional">> => false, |
| 6 | - <<"requirement">> => <<"1.0.0">>}, |
| 6 | + <<"requirement">> => <<"1.1.0">>}, |
| 7 7 | <<"supervisor3">> => #{<<"app">> => <<"supervisor3">>, |
| 8 8 | <<"optional">> => false, |
| 9 9 | <<"requirement">> => <<"1.1.5">>}}}. |
| @@ -29,7 +29,8 @@ | |
| 29 29 | <<"src/brod_cli_pipe.erl">>,<<"src/brod_client.erl">>, |
| 30 30 | <<"src/brod_consumer.erl">>,<<"src/brod_consumers_sup.erl">>, |
| 31 31 | <<"src/brod_group_coordinator.erl">>,<<"src/brod_group_member.erl">>, |
| 32 | - <<"src/brod_group_subscriber.erl">>,<<"src/brod_kafka_requests.erl">>, |
| 32 | + <<"src/brod_group_subscriber.erl">>,<<"src/brod_kafka_apis.erl">>, |
| 33 | + <<"src/brod_kafka_request.erl">>,<<"src/brod_kafka_requests.erl">>, |
| 33 34 | <<"src/brod_producer.erl">>,<<"src/brod_producer_buffer.erl">>, |
| 34 35 | <<"src/brod_producers_sup.erl">>,<<"src/brod_sock.erl">>, |
| 35 36 | <<"src/brod_sup.erl">>,<<"src/brod_topic_subscriber.erl">>, |
| @@ -45,6 +45,9 @@ | |
| 45 45 | %% Is kafka error code |
| 46 46 | -define(IS_ERROR(EC), kpro_error_code:is_error(EC)). |
| 47 47 | |
| 48 | + -define(KV(Key, Value), {Key, Value}). |
| 49 | + -define(TKV(Ts, Key, Value), {Ts, Key, Value}). |
| 50 | + |
| 48 51 | -endif. % include brod_int.hrl |
| 49 52 | |
| 50 53 | %%%_* Emacs ==================================================================== |
| @@ -1,4 +1,4 @@ | |
| 1 1 | {deps, [ {supervisor3, "1.1.5"} |
| 2 | - , {kafka_protocol, "1.0.0"} |
| 2 | + , {kafka_protocol, "1.1.0"} |
| 3 3 | ]}. |
| 4 4 | {erl_opts, [warn_unused_vars,warn_shadow_vars,warn_unused_import,warn_obsolete_guard,debug_info]}. |
Loading more files…