现在位置: 首页 / Kafka / kafka / 正文

Kafka 性能测试

  • 桩白墨
  • Kafka
  • 2020-10-23
  • 188热度
  • 0评论

接触 kafka 有一段时间了，突然想看下 kafka 的性能到底怎么样，搜了相关资料，发现它自带了性能测试脚本。记录一下全过程。

1 本地搭建 kafka 环境

使用 docker-compose。https://github.com/plough/docker-compose/tree/master/kafka

启动之后，docker exec -it xxx bash 进入 kafka 的容器，cd 到 /opt/kafka_2.12-2.2.0/bin，准备开始。

2 创建测试用的 topic

./kafka-topics.sh --create --zookeeper zookeeper:2181 --topic test_producer_perf --partitions 2 --replication-factor 1

3 生产者性能测试

./kafka-producer-perf-test.sh --topic test_producer_perf --num-records 5000000 --throughput 100000 --producer-props bootstrap.servers=zookeeper:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 10

参数解读：

  • --num-records: 生产多少条消息
  • --throughput: 每秒生产多少条（生产速度）
  • --record-size: 每条消息多大（单位：字节）

运行结果记录：

bash-4.4# ./kafka-producer-perf-test.sh --topic test_producer_perf --num-records 500000 --throughput 10 --producer-props bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 1
52 records sent, 10.4 records/sec (0.00 MB/sec), 12.4 ms avg latency, 237.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 4.6 ms avg latency, 8.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 4.1 ms avg latency, 9.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 3.5 ms avg latency, 6.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 2.0 ms avg latency, 6.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 3.0 ms avg latency, 5.0 ms max latency.
51 records sent, 10.2 records/sec (0.00 MB/sec), 2.1 ms avg latency, 5.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 3.1 ms avg latency, 6.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 2.6 ms avg latency, 5.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 2.8 ms avg latency, 5.0 ms max latency.
50 records sent, 10.0 records/sec (0.00 MB/sec), 2.5 ms avg latency, 5.0 ms max latency.

bash-4.4# ./kafka-producer-perf-test.sh --topic test_producer_perf --num-records 500000 --throughput 100000 --producer-props bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 1
499876 records sent, 99975.2 records/sec (0.10 MB/sec), 2.5 ms avg latency, 159.0 ms max latency.
500000 records sent, 99760.574621 records/sec (0.10 MB/sec), 2.52 ms avg latency, 159.00 ms max latency, 1 ms 50th, 10 ms 95th, 21 ms 99th, 27 ms 99.9th.

bash-4.4# ./kafka-producer-perf-test.sh --topic test_producer_perf --num-records 500000 --throughput 100000 --producer-props bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 10
499902 records sent, 99940.4 records/sec (0.95 MB/sec), 1.5 ms avg latency, 159.0 ms max latency.
500000 records sent, 99800.399202 records/sec (0.95 MB/sec), 1.47 ms avg latency, 159.00 ms max latency, 1 ms 50th, 5 ms 95th, 9 ms 99th, 15 ms 99.9th.

bash-4.4# ./kafka-producer-perf-test.sh --topic test_producer_perf --num-records 5000000 --throughput 100000 --producer-props bootstrap.servers=localhost:9092 key.serializer=org.apache.kafka.common.serialization.StringSerializer value.serializer=org.apache.kafka.common.serialization.StringSerializer --record-size 10
486002 records sent, 96929.0 records/sec (0.92 MB/sec), 1.6 ms avg latency, 169.0 ms max latency.
515447 records sent, 103089.4 records/sec (0.98 MB/sec), 4.1 ms avg latency, 153.0 ms max latency.
500653 records sent, 100110.6 records/sec (0.95 MB/sec), 1.2 ms avg latency, 31.0 ms max latency.
499881 records sent, 99976.2 records/sec (0.95 MB/sec), 1.5 ms avg latency, 19.0 ms max latency.
500354 records sent, 100070.8 records/sec (0.95 MB/sec), 1.9 ms avg latency, 14.0 ms max latency.
499865 records sent, 99973.0 records/sec (0.95 MB/sec), 2.0 ms avg latency, 42.0 ms max latency.
500200 records sent, 100040.0 records/sec (0.95 MB/sec), 1.8 ms avg latency, 19.0 ms max latency.
500100 records sent, 100020.0 records/sec (0.95 MB/sec), 1.6 ms avg latency, 16.0 ms max latency.
500113 records sent, 100022.6 records/sec (0.95 MB/sec), 1.9 ms avg latency, 16.0 ms max latency.
5000000 records sent, 99938.038416 records/sec (0.95 MB/sec), 1.98 ms avg latency, 169.00 ms max latency, 1 ms 50th, 5 ms 95th, 13 ms 99th, 64 ms 99.9th.

4 消费者性能测试

./kafka-consumer-perf-test.sh --topic test_producer_perf  --messages 5000000 --fetch-size 1000000 --broker-list=localhost:9092

参数解读：

  • --messages: 一共拉取多少条消息
  • --fetch-size: 单次请求拉取多少条消息

运行结果记录：

bash-4.4# ./kafka-consumer-perf-test.sh --topic test_producer_perf  --messages 5000000 --fetch-size 100000 --broker-list=localhost:9092 
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-23 13:09:57:437, 2020-10-23 13:10:00:426, 43.3881, 14.5159, 5000074, 1672825.0251, 82, 2907, 14.9254, 1720011.6959

bash-4.4# ./kafka-consumer-perf-test.sh --topic test_producer_perf  --messages 5000000 --fetch-size 10000 --broker-list=localhost:9092 
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-23 13:15:48:738, 2020-10-23 13:15:55:002, 43.3899, 6.9269, 5000264, 798254.1507, 15, 6249, 6.9435, 800170.2672

bash-4.4# ./kafka-consumer-perf-test.sh --topic test_producer_perf  --messages 5000000 --fetch-size 1000000 --broker-list=localhost:9092 
start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec
2020-10-23 13:16:12:706, 2020-10-23 13:16:14:247, 43.3873, 28.1553, 5000000, 3244646.3335, 12, 1529, 28.3763, 3270111.1838

5 小结

从测试结果来看：

  • 写入速度在 10w条/s 以上（因为测试时速度限定了 10w/s，实际的上限可能远大于这个数字）
  • 读取速度达到了 300w条/s

可以得出一个粗略的结论：单机 kafka，吞吐量超过 10w/s。kafka 的性能的确非常强劲，大多数情况下可以放心使用。

更进一步

有时间可以继续探究：

  • 找出写入速度的极限值
  • 变换单条记录的大小
  • 以 MB/s 为单位测试吞吐量
  • 开启多线程
  • 测试 kafka 集群的性能