Packages
brod
3.9.0
4.5.7
4.5.6
4.5.5
4.5.4
4.5.3
4.5.2
4.5.1
4.5.0
4.4.7
4.4.6
4.4.5
4.4.4
4.4.3
4.4.2
4.4.1
4.4.0
4.3.3
4.3.2
4.3.1
4.3.0
4.2.0
4.1.1
4.1.0
4.0.0
3.19.1
3.19.0
3.18.0
3.17.1
3.17.0
3.16.5
3.16.4
3.16.3
3.16.2
3.16.1
3.16.0
3.15.6
3.15.5
3.15.4
3.15.3
3.15.1
3.15.0
3.14.0
3.13.0
3.12.0
3.11.0
3.10.0
3.9.5
3.9.3
3.9.2
3.9.1
3.9.0
3.8.1
3.8.0
3.7.11
3.7.10
3.7.9
3.7.8
3.7.7
3.7.6
3.7.5
3.7.4
3.7.3
3.7.2
3.7.1
3.7.0
3.6.2
3.6.1
3.6.0
3.5.2
3.5.1
3.5.0
3.4.0
3.3.5
3.3.4
3.3.3
3.3.2
3.3.1
3.3.0
3.2.0
3.0.0
2.5.0
2.4.1
2.4.0
2.3.7
2.3.6
2.3.5
2.3.4
2.3.3
2.3.1
2.2.16
2.2.15
2.2.14
2.2.12
2.2.11
2.2.10
2.2.9
2.2.8
2.2.7
2.2.6
2.2.5
2.2.4
2.2.3
2.2.2
2.2.1
2.2.0
2.1.12
2.1.11
2.1.10
2.1.8
2.1.7
2.1.4
2.1.2
2.0.0
Apache Kafka Erlang client library
Current section
113 Versions
Jump to
Current section
113 Versions
Compare versions
37
files changed
+10855
additions
-65
deletions
| @@ -1,14 +1,14 @@ | |
| 1 1 | KAFKA_VERSION ?= 2.2 |
| 2 2 | PROJECT = brod |
| 3 3 | PROJECT_DESCRIPTION = Kafka client library in Erlang |
| 4 | - PROJECT_VERSION = 3.8.1 |
| 4 | + PROJECT_VERSION = 3.9.0 |
| 5 5 | |
| 6 6 | DEPS = supervisor3 kafka_protocol |
| 7 7 | |
| 8 8 | ERLC_OPTS = -Werror +warn_unused_vars +warn_shadow_vars +warn_unused_import +warn_obsolete_guard +debug_info -Dbuild_brod_cli |
| 9 9 | |
| 10 10 | dep_supervisor3_commit = 1.1.8 |
| 11 | - dep_kafka_protocol_commit = 2.2.9 |
| 11 | + dep_kafka_protocol_commit = 2.3.1 |
| 12 12 | dep_kafka_protocol = git https://github.com/klarna/kafka_protocol.git $(dep_kafka_protocol_commit) |
| 13 13 | |
| 14 14 | EDOC_OPTS = preprocess, {macros, [{build_brod_cli, true}]} |
| @@ -45,9 +45,10 @@ rebar-clean: | |
| 45 45 | |
| 46 46 | hex-publish: distclean |
| 47 47 | @rebar3 hex publish |
| 48 | + @rebar3 hex docs |
| 48 49 | |
| 49 50 | ## tests that require kafka running at localhost |
| 50 | - INTEGRATION_CTS = brod_cg_commits brod_client brod_compression brod_consumer brod_producer brod_group_subscriber brod_topic_subscriber |
| 51 | + INTEGRATION_CTS = brod_cg_commits brod_client brod_compression brod_consumer brod_producer brod_group_subscriber brod_topic_subscriber brod |
| 51 52 | |
| 52 53 | ## integration tests |
| 53 54 | it: ut $(INTEGRATION_CTS:%=it-%) |
| @@ -29,7 +29,7 @@ Why "brod"? [http://en.wikipedia.org/wiki/Max_Brod](http://en.wikipedia.org/wiki | |
| 29 29 | # Working With Kafka 0.9.x or Earlier |
| 30 30 | |
| 31 31 | Make sure `{query_api_versions, false}` exists in client config. |
| 32 | - This is because `ApiVersionRequest` was introduced in kafka 0.10, |
| 32 | + This is because `ApiVersionRequest` was introduced in kafka 0.10, |
| 33 33 | sending such request to older version brokers will cause connection failure. |
| 34 34 | |
| 35 35 | e.g. in sys.config: |
| @@ -95,14 +95,14 @@ Brod supervision (and process link) tree. | |
| 95 95 | |
| 96 96 | # Clients |
| 97 97 | |
| 98 | - A `brod_client` in brod is a `gen_server` responsible for establishing and |
| 99 | - maintaining tcp sockets connecting to kafka brokers. |
| 100 | - It also manages per-topic-partition producer and consumer processes under |
| 98 | + A `brod_client` in brod is a `gen_server` responsible for establishing and |
| 99 | + maintaining tcp sockets connecting to kafka brokers. |
| 100 | + It also manages per-topic-partition producer and consumer processes under |
| 101 101 | two-level supervision trees. |
| 102 102 | |
| 103 103 | ## Start clients by default |
| 104 104 | |
| 105 | - You may include client configs in sys.config have them started by default |
| 105 | + You may include client configs in sys.config have them started by default |
| 106 106 | (by application controller) |
| 107 107 | |
| 108 108 | Example of configuration (for sys.config): |
| @@ -125,7 +125,7 @@ Example of configuration (for sys.config): | |
| 125 125 | |
| 126 126 | ## Start brod client on demand |
| 127 127 | |
| 128 | - You may also call `brod:start_client/1,2,3` to start a client on demand, |
| 128 | + You may also call `brod:start_client/1,2,3` to start a client on demand, |
| 129 129 | which will be added to brod supervision tree. |
| 130 130 | |
| 131 131 | ```erlang |
| @@ -239,8 +239,8 @@ brod:produce(_Client = brod_client_1, | |
| 239 239 | |
| 240 240 | ## Handle Acks from Kafka as Messages |
| 241 241 | |
| 242 | - For async produce APIs `brod:produce/3` and `brod:produce/5`, |
| 243 | - the caller should expect a message of below pattern for each produce call. |
| 242 | + For async produce APIs `brod:produce/3` and `brod:produce/5`, |
| 243 | + the caller should expect a message of below pattern for each produce call. |
| 244 244 | |
| 245 245 | ```erlang |
| 246 246 | #brod_produce_reply{ call_ref = CallRef %% returned from brod:produce |
| @@ -249,84 +249,84 @@ the caller should expect a message of below pattern for each produce call. | |
| 249 249 | ``` |
| 250 250 | Add `-include_lib("brod/include/brod.hrl").` to use the record. |
| 251 251 | |
| 252 | - In case the `brod:produce` caller is a process like `gen_server` which |
| 253 | - receives ALL messages, the callers should keep the call references in its |
| 254 | - looping state and match the replies against them when received. |
| 252 | + In case the `brod:produce` caller is a process like `gen_server` which |
| 253 | + receives ALL messages, the callers should keep the call references in its |
| 254 | + looping state and match the replies against them when received. |
| 255 255 | Otherwise `brod:sync_produce_request/1` can be used to block-wait for acks. |
| 256 256 | |
| 257 | - NOTE: If `required_acks` is set to `none` in producer config, |
| 258 | - kafka will NOT ack the requests, and the reply message is sent back |
| 257 | + NOTE: If `required_acks` is set to `none` in producer config, |
| 258 | + kafka will NOT ack the requests, and the reply message is sent back |
| 259 259 | to caller immediately after the message has been sent to the socket process. |
| 260 260 | |
| 261 | - NOTE: The replies are only strictly ordered per-partition. |
| 262 | - i.e. if the caller is producing to two or more partitions, |
| 263 | - it may receive replies ordered differently than in which order |
| 261 | + NOTE: The replies are only strictly ordered per-partition. |
| 262 | + i.e. if the caller is producing to two or more partitions, |
| 263 | + it may receive replies ordered differently than in which order |
| 264 264 | `brod:produce` API was called. |
| 265 265 | |
| 266 266 | ## Handle Acks from Kafka in Callback Function |
| 267 267 | |
| 268 | - Async APIs `brod:produce_cb/4` and `brod:produce_cb/6` allow callers to |
| 269 | - provided a callback function to handle acknowledgements from kafka. |
| 270 | - In this case, the caller may want to monitor the producer process because |
| 268 | + Async APIs `brod:produce_cb/4` and `brod:produce_cb/6` allow callers to |
| 269 | + provided a callback function to handle acknowledgements from kafka. |
| 270 | + In this case, the caller may want to monitor the producer process because |
| 271 271 | then they know that the callbacks will not be evaluated if the producer is 'DOWN', |
| 272 272 | and there is perhaps a need for retry. |
| 273 273 | |
| 274 274 | # Consumers |
| 275 275 | |
| 276 | - Kafka consumers work in poll mode. In brod, `brod_consumer` is the poller, |
| 277 | - which is constantly asking for more data from the kafka node which is a leader |
| 276 | + Kafka consumers work in poll mode. In brod, `brod_consumer` is the poller, |
| 277 | + which is constantly asking for more data from the kafka node which is a leader |
| 278 278 | for the given partition. |
| 279 279 | |
| 280 | - By subscribing to `brod_consumer` a process should receive the polled message |
| 280 | + By subscribing to `brod_consumer` a process should receive the polled message |
| 281 281 | sets (not individual messages) into its mailbox. |
| 282 282 | |
| 283 | - In brod, we have so far implemented two different subscribers |
| 284 | - (`brod_topic_subscriber` and `brod_group_subscriber`), |
| 283 | + In brod, we have so far implemented two different subscribers |
| 284 | + (`brod_topic_subscriber` and `brod_group_subscriber`), |
| 285 285 | hopefully covered most of the common use cases. |
| 286 286 | |
| 287 | - For maximum flexibility, an applications may implement their own |
| 287 | + For maximum flexibility, an applications may implement their own |
| 288 288 | per-partition subscriber. |
| 289 289 | |
| 290 | - Below diagrams illustrate 3 examples of how subscriber processes may work |
| 290 | + Below diagrams illustrate 3 examples of how subscriber processes may work |
| 291 291 | with `brod_consumer`. |
| 292 292 | |
| 293 293 | ## Partition subscriber |
| 294 294 |  |
| 295 295 | |
| 296 | - This gives the best flexibility as the per-partition subscribers work |
| 296 | + This gives the best flexibility as the per-partition subscribers work |
| 297 297 | directly with per-partition pollers. |
| 298 298 | |
| 299 | - The messages are delivered to subscribers in message sets (batches), |
| 300 | - not individual messages, (however the subscribers are allowed to |
| 299 | + The messages are delivered to subscribers in message sets (batches), |
| 300 | + not individual messages, (however the subscribers are allowed to |
| 301 301 | ack individual offsets). |
| 302 302 | |
| 303 303 | ## Topic subscriber (`brod_topic_subscriber`) |
| 304 304 |  |
| 305 305 | |
| 306 | - A topic subscriber provides the easiest way to receive and process |
| 307 | - messages from ALL partitions of a given topic. See `brod_demo_cg_collector` |
| 306 | + A topic subscriber provides the easiest way to receive and process |
| 307 | + messages from ALL partitions of a given topic. See `brod_demo_cg_collector` |
| 308 308 | and `brod_demo_topic_subscriber` for example. |
| 309 309 | |
| 310 | - Users may choose to implement the `brod_topic_subscriber` behaviour callbacks |
| 311 | - in a module, or simply provide an anonymous callback function to have the |
| 310 | + Users may choose to implement the `brod_topic_subscriber` behaviour callbacks |
| 311 | + in a module, or simply provide an anonymous callback function to have the |
| 312 312 | individual messages processed. |
| 313 313 | |
| 314 314 | ## Group subscriber (`brod_group_subscriber`) |
| 315 315 |  |
| 316 316 | |
| 317 | - Similar to topic subscriber, the `brod_group_subscriber` behaviour callbacks |
| 318 | - are to be implemented to process individual messages. See |
| 319 | - `brod_demo_group_subscriber_koc` and `brod_demo_group_subscriber_loc` |
| 317 | + Similar to topic subscriber, the `brod_group_subscriber` behaviour callbacks |
| 318 | + are to be implemented to process individual messages. See |
| 319 | + `brod_demo_group_subscriber_koc` and `brod_demo_group_subscriber_loc` |
| 320 320 | for example. |
| 321 321 | |
| 322 | - A group subscriber is started by giving a set of topics, some |
| 323 | - (maybe none, or maybe all) of the partitions in the topic set will be |
| 324 | - assigned to it, then the subscriber should subscribe to ALL the assigned |
| 322 | + A group subscriber is started by giving a set of topics, some |
| 323 | + (maybe none, or maybe all) of the partitions in the topic set will be |
| 324 | + assigned to it, then the subscriber should subscribe to ALL the assigned |
| 325 325 | partitions. |
| 326 326 | |
| 327 | - Users may also choose to implement the `brod_group_member` behaviour (callbacks |
| 328 | - for `brod_group_coordinator`) for a different group subscriber (e.g. spawn |
| 329 | - one subscriber per partition), see [brucke](https://github.com/klarna/brucke) |
| 327 | + Users may also choose to implement the `brod_group_member` behaviour (callbacks |
| 328 | + for `brod_group_coordinator`) for a different group subscriber (e.g. spawn |
| 329 | + one subscriber per partition), see [brucke](https://github.com/klarna/brucke) |
| 330 330 | for example. |
| 331 331 | |
| 332 332 | ### Example of group consumer which commits offsets to Kafka |
| @@ -405,11 +405,25 @@ await response and then close the connection. | |
| 405 405 | Hosts = [{"localhost", 9092}]. |
| 406 406 | Topic = <<"topic">>. |
| 407 407 | Partition = 0. |
| 408 | + Timeout = 1000. |
| 409 | + TopicConfigs = [ |
| 410 | + #{ |
| 411 | + config_entries => [], |
| 412 | + num_partitions => 1, |
| 413 | + replica_assignment => [], |
| 414 | + replication_factor => 1, |
| 415 | + topic => Topic |
| 416 | + } |
| 417 | + ] |
| 408 418 | brod:get_metadata(Hosts). |
| 419 | + brod:create_topics(Hosts, TopicConfigs, #{timeout => Timeout}). |
| 409 420 | brod:get_metadata(Hosts, [Topic]). |
| 421 | + brod:delete_topics(Hosts, [Topic], Timeout). |
| 410 422 | brod:resolve_offset(Hosts, Topic, Partition). |
| 411 423 | ``` |
| 412 424 | |
| 425 | + Caution the above delete_topics can fail if you do not have `delete.topic.enable` set to true in your kafka config |
| 426 | + |
| 413 427 | # brod-cli: A command line tool to interact with Kafka |
| 414 428 | |
| 415 429 | This will build a self-contained binary with brod application |
| @@ -507,6 +521,4 @@ brod commits -b localhost:9092 -i the-group-id -t topic-name -o "0:10000" --prot | |
| 507 521 | * HTML tagged EDoc |
| 508 522 | * Support scram-sasl in brod-cli |
| 509 523 | * lz4 compression & decompression |
| 510 | - * Create/delete topic APIs |
| 511 524 | * Transactional produce APIs |
| 512 | - |
| @@ -1,9 +1,9 @@ | |
| 1 1 | {<<"name">>,<<"brod">>}. |
| 2 | - {<<"version">>,<<"3.8.1">>}. |
| 2 | + {<<"version">>,<<"3.9.0">>}. |
| 3 3 | {<<"requirements">>, |
| 4 4 | #{<<"kafka_protocol">> => |
| 5 5 | #{<<"app">> => <<"kafka_protocol">>,<<"optional">> => false, |
| 6 | - <<"requirement">> => <<"2.2.9">>}, |
| 6 | + <<"requirement">> => <<"2.3.1">>}, |
| 7 7 | <<"supervisor3">> => |
| 8 8 | #{<<"app">> => <<"supervisor3">>,<<"optional">> => false, |
| 9 9 | <<"requirement">> => <<"1.1.8">>}}}. |
| @@ -15,11 +15,33 @@ | |
| 15 15 | [<<"src/brod.app.src">>,<<"LICENSE">>,<<"Makefile">>,<<"NOTICE">>, |
| 16 16 | <<"README.md">>,<<"erlang.mk">>,<<"include/brod.hrl">>, |
| 17 17 | <<"include/brod_int.hrl">>,<<"rebar.config">>,<<"rebar.config.script">>, |
| 18 | - <<"src/brod.erl">>,<<"src/brod_cg_commits.erl">>,<<"src/brod_cli.erl">>, |
| 18 | + <<"src/_build/test/extras/.rebar3/erlcinfo">>, |
| 19 | + <<"src/_build/test/extras/brod.erl">>, |
| 20 | + <<"src/_build/test/extras/brod_cg_commits.erl">>, |
| 21 | + <<"src/_build/test/extras/brod_cli.erl">>, |
| 22 | + <<"src/_build/test/extras/brod_cli_pipe.erl">>, |
| 23 | + <<"src/_build/test/extras/brod_client.erl">>, |
| 24 | + <<"src/_build/test/extras/brod_consumer.erl">>, |
| 25 | + <<"src/_build/test/extras/brod_consumers_sup.erl">>, |
| 26 | + <<"src/_build/test/extras/brod_group_coordinator.erl">>, |
| 27 | + <<"src/_build/test/extras/brod_group_member.erl">>, |
| 28 | + <<"src/_build/test/extras/brod_group_subscriber.erl">>, |
| 29 | + <<"src/_build/test/extras/brod_group_subscriber_v2.erl">>, |
| 30 | + <<"src/_build/test/extras/brod_group_subscriber_worker.erl">>, |
| 31 | + <<"src/_build/test/extras/brod_kafka_apis.erl">>, |
| 32 | + <<"src/_build/test/extras/brod_kafka_request.erl">>, |
| 33 | + <<"src/_build/test/extras/brod_producer.erl">>, |
| 34 | + <<"src/_build/test/extras/brod_producer_buffer.erl">>, |
| 35 | + <<"src/_build/test/extras/brod_producers_sup.erl">>, |
| 36 | + <<"src/_build/test/extras/brod_sup.erl">>, |
| 37 | + <<"src/_build/test/extras/brod_topic_subscriber.erl">>, |
| 38 | + <<"src/_build/test/extras/brod_utils.erl">>,<<"src/brod.erl">>, |
| 39 | + <<"src/brod_cg_commits.erl">>,<<"src/brod_cli.erl">>, |
| 19 40 | <<"src/brod_cli_pipe.erl">>,<<"src/brod_client.erl">>, |
| 20 41 | <<"src/brod_consumer.erl">>,<<"src/brod_consumers_sup.erl">>, |
| 21 42 | <<"src/brod_group_coordinator.erl">>,<<"src/brod_group_member.erl">>, |
| 22 | - <<"src/brod_group_subscriber.erl">>,<<"src/brod_kafka_apis.erl">>, |
| 43 | + <<"src/brod_group_subscriber.erl">>,<<"src/brod_group_subscriber_v2.erl">>, |
| 44 | + <<"src/brod_group_subscriber_worker.erl">>,<<"src/brod_kafka_apis.erl">>, |
| 23 45 | <<"src/brod_kafka_request.erl">>,<<"src/brod_producer.erl">>, |
| 24 46 | <<"src/brod_producer_buffer.erl">>,<<"src/brod_producers_sup.erl">>, |
| 25 47 | <<"src/brod_sup.erl">>,<<"src/brod_topic_subscriber.erl">>, |
| @@ -1,5 +1,5 @@ | |
| 1 1 | {deps, [ {supervisor3, "1.1.8"} |
| 2 | - , {kafka_protocol, "2.2.9"} |
| 2 | + , {kafka_protocol, "2.3.1"} |
| 3 3 | ]}. |
| 4 4 | {edoc_opts, [{preprocess, true}, {macros, [{build_brod_cli, true}]}]}. |
| 5 5 | {erl_opts, [warn_unused_vars,warn_shadow_vars,warn_unused_import,warn_obsolete_guard,debug_info]}. |
| @@ -9,6 +9,8 @@ | |
| 9 9 | |
| 10 10 | {escript_incl_apps, [docopt, brod]}. |
| 11 11 | |
| 12 | + {ct_opts, [{enable_builtin_hooks, false}]}. |
| 13 | + |
| 12 14 | {cover_enabled, true}. |
| 13 15 | {cover_opts, [verbose]}. |
| 14 16 | {cover_export_enabled, true}. |
| @@ -2,6 +2,7 @@ IsRebar3 = erlang:function_exported(rebar3, main, 1), | |
| 2 2 | DocoptUrl = "https://github.com/zmstone/docopt-erl.git", |
| 3 3 | DocOptTag = "0.1.3", |
| 4 4 | DocoptDep = {docopt, {git, DocoptUrl, {branch, DocOptTag}}}, |
| 5 | + Snabbkaffe = {snabbkaffe, {git, "https://github.com/klarna/snabbkaffe.git", {branch, "master"}}}, % TODO: use release tag |
| 5 6 | Profiles = |
| 6 7 | {profiles, [ |
| 7 8 | {brod_cli, [ |
| @@ -17,7 +18,7 @@ Profiles = | |
| 17 18 | ]} |
| 18 19 | ]}]}, |
| 19 20 | {test, [ |
| 20 | - {deps, [meck, proper, jsone, DocoptDep]}, |
| 21 | + {deps, [meck, proper, jsone, DocoptDep, Snabbkaffe]}, |
| 21 22 | {erl_opts, [{d, build_brod_cli}]} |
| 22 23 | ]} |
| 23 24 | ]}, |
| @@ -49,4 +50,3 @@ case os:getenv("TRAVIS") of | |
| 49 50 | _ -> |
| 50 51 | CONFIG1 |
| 51 52 | end. |
| 52 | - |
Loading more files…