2020年10月23日

Kafka 性能测试

接触 kafka 有一段时间了,突然想看下 kafka 的性能到底怎么样,搜了相关资料,发现它自带了性能测试脚本。记录一下全过程。

1 本地搭建 kafka 环境

使用 docker-compose。https://github.com/plough/docker-compose/tree/master/kafka

启动之后,docker exec -it xxx bash 进入 kafka 的容器,cd 到 /opt/kafka_2.12-2.2.0/bin,准备开始。

2 创建测试用的 topic

./kafka-topics.sh --create --zookeeper zookeeper:2181 --topic test_producer_perf --partitions 2 --replication-factor 1

3 生产者性能测试

./kafka-producer-perf-test.sh --topic test_producer_perf --num-records 5000000 --throughput 100000 --producer-props bootstrap.servers=zookeeper:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 10

参数解读:

  • –num-records: 生产多少条消息
  • –throughput: 每秒生产多少条(生产速度)
  • –record-size: 每条消息多大(单位:字节)

运行结果记录:

bash-4.4# ./kafka-producer-perf-test.sh --topic test_producer_perf --num-records 500000 --throughput 10 --producer-props bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 1
52 records sent, 10.4 records/sec (0.00 MB/sec), 12.4 ms avg latency, 237.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 4.6 ms avg latency, 8.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 4.1 ms avg latency, 9.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 3.5 ms avg latency, 6.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 2.0 ms avg latency, 6.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 3.0 ms avg latency, 5.0 ms max latency.
51 records sent, 10.2 records/sec (0.00 MB/sec), 2.1 ms avg latency, 5.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 3.1 ms avg latency, 6.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 2.6 ms avg latency, 5.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 2.8 ms avg latency, 5.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 2.5 ms avg latency, 5.0 ms max latency.

bash-4.4# ./kafka-producer-perf-test.sh --topic test_producer_perf --num-records 500000 --throughput 100000 --producer-props bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 1
499876 records sent, 99975.2 records/sec (0.10 MB/sec), 2.5 ms avg latency, 159.0 ms max latency.
500000 records sent, 99760.574621 records/sec (0.10 MB/sec), 2.52 ms avg latency, 159.00 ms max latency, 1 ms 50th, 10 ms 95th, 21 ms 99th, 27 ms 99.9th.

bash-4.4# ./kafka-producer-perf-test.sh --topic test_producer_perf --num-records 500000 --throughput 100000 --producer-props bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 10
499902 records sent, 99940.4 records/sec (0.95 MB/sec), 1.5 ms avg latency, 159.0 ms max latency.
500000 records sent, 99800.399202 records/sec (0.95 MB/sec), 1.47 ms avg latency, 159.00 ms max latency, 1 ms 50th, 5 ms 95th, 9 ms 99th, 15 ms 99.9th.

bash-4.4# ./kafka-producer-perf-test.sh --topic test_producer_perf --num-records 5000000 --throughput 100000 --producer-props bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 10
486002 records sent, 96929.0 records/sec (0.92 MB/sec), 1.6 ms avg latency, 169.0 ms max latency.
515447 records sent, 103089.4 records/sec (0.98 MB/sec), 4.1 ms avg latency, 153.0 ms max latency.
500653 records sent, 100110.6 records/sec (0.95 MB/sec), 1.2 ms avg latency, 31.0 ms max latency.
499881 records sent, 99976.2 records/sec (0.95 MB/sec), 1.5 ms avg latency, 19.0 ms max latency.
500354 records sent, 100070.8 records/sec (0.95 MB/sec), 1.9 ms avg latency, 14.0 ms max latency.
499865 records sent, 99973.0 records/sec (0.95 MB/sec), 2.0 ms avg latency, 42.0 ms max latency.
500200 records sent, 100040.0 records/sec (0.95 MB/sec), 1.8 ms avg latency, 19.0 ms max latency.
500100 records sent, 100020.0 records/sec (0.95 MB/sec), 1.6 ms avg latency, 16.0 ms max latency.
500113 records sent, 100022.6 records/sec (0.95 MB/sec), 1.9 ms avg latency, 16.0 ms max latency.
5000000 records sent, 99938.038416 records/sec (0.95 MB/sec), 1.98 ms avg latency, 169.00 ms max latency, 1 ms 50th, 5 ms 95th, 13 ms 99th, 64 ms 99.9th.

4 消费者性能测试

./kafka-consumer-perf-test.sh --topic test_producer_perf  --messages 5000000 --fetch-size 1000000 --broker-list=localhost:9092

参数解读:

  • –messages: 一共拉取多少条消息
  • –fetch-size: 单次请求拉取多少条消息

运行结果记录:

bash-4.4# ./kafka-consumer-perf-test.sh --topic test_producer_perf  --messages 5000000 --fetch-size 100000 --broker-list=localhost:9092 
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-23 13:09:57:437, 2020-10-23 13:10:00:426, 43.3881, 14.5159, 5000074, 1672825.0251, 82, 2907, 14.9254, 1720011.6959

bash-4.4# ./kafka-consumer-perf-test.sh --topic test_producer_perf  --messages 5000000 --fetch-size 10000 --broker-list=localhost:9092 
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-23 13:15:48:738, 2020-10-23 13:15:55:002, 43.3899, 6.9269, 5000264, 798254.1507, 15, 6249, 6.9435, 800170.2672

bash-4.4# ./kafka-consumer-perf-test.sh --topic test_producer_perf  --messages 5000000 --fetch-size 1000000 --broker-list=localhost:9092 
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-23 13:16:12:706, 2020-10-23 13:16:14:247, 43.3873, 28.1553, 5000000, 3244646.3335, 12, 1529, 28.3763, 3270111.1838

5 小结

从测试结果来看:

  • 写入速度在 10w条/s 以上(因为测试时速度限定了 10w/s,实际的上限可能远大于这个数字)
  • 读取速度达到了 300w条/s

可以得出一个粗略的结论:单机 kafka,吞吐量超过 10w/s。kafka 的性能的确非常强劲,大多数情况下可以放心使用。

更进一步

有时间可以继续探究:

  • 找出写入速度的极限值
  • 变换单条记录的大小
  • 以 MB/s 为单位测试吞吐量
  • 开启多线程
  • 测试 kafka 集群的性能

You may also like...

发表评论

电子邮件地址不会被公开。 必填项已用*标注


*