23:39:48.928 [INFO - main] (KuduTestHarness.java:155) Creating a new MiniKuduCluster...
23:39:48.928 [INFO - main] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-tasksngkcS/test-tmp
23:39:48.928 [INFO - main] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
23:39:48.928 [INFO - main] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-tasksngkcS/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
23:39:48.930 [DEBUG - main] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 3
enable_kerberos: false
cluster_root: "/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997"
hms_mode: NONE
mini_kdc_options {
}
principal: "kudu"
mini_oidc_options {
}
}
23:39:49.259 [DEBUG - main] (MiniKuduCluster.java:195) Response:
23:39:49.260 [DEBUG - main] (MiniKuduCluster.java:185) Request: start_cluster {
}
23:39:49.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:49.262970 6456 mini_oidc.cc:152] Starting JWKS server
23:39:49.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:49.267190 6456 webserver.cc:460] Webserver started at http://127.0.0.1:37281/ using document root <none> and password file <none>
23:39:49.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:49.267617 6456 mini_oidc.cc:189] Starting OIDC Discovery server
23:39:49.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:49.269397 6456 webserver.cc:460] Webserver started at http://127.0.0.1:43049/ using document root <none> and password file <none>
23:39:49.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-10T23:39:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
23:39:49.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-10T23:39:49Z Disabled control of system clock
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:49.325902 6456 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/wal
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/data
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/logs
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/data/info.pb
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.62:46719
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.62
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:39:49.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37281/jwks.json with env {}
23:39:49.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.637010 6469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:39:49.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.637567 6469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:39:49.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.638056 6469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:39:49.669 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.669623 6469 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
23:39:49.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.669929 6469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:39:49.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.670138 6469 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
23:39:49.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.670408 6469 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
23:39:49.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.670619 6469 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
23:39:49.670 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.670765 6469 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37281/jwks.json
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:49.706744 6469 master_runner.cc:386] Master server non-default flags:
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/data
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/wal
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.62:46719
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37281/jwks.json
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/data/info.pb
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.62
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/logs
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:39:49.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:39:49.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:49.707968 6469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:39:49.709 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:49.709524 6469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:39:49.728 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.728032 6482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:50.931 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.728050 6478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:50.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:49.730610 6469 server_base.cc:1034] running on GCE node
23:39:50.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:49.728065 6477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:50.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.935945 6469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:39:50.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.938706 6469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:39:50.940 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.940095 6469 hybrid_clock.cc:648] HybridClock initialized: now 1786405190940067 us; error 35 us; skew 500 ppm
23:39:50.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.940989 6469 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:39:50.951 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.950997 6469 webserver.cc:460] Webserver started at http://127.6.78.62:45937/ using document root <none> and password file <none>
23:39:50.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.951987 6469 fs_manager.cc:359] Metadata directory not provided
23:39:50.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.952236 6469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:39:50.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.952756 6469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
23:39:50.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.957746 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/data/instance:
23:39:50.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b96c54d5bac944588ef07845f2c6874e"
23:39:50.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:50 on dist-test-slave-9jjr"
23:39:50.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.958961 6469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/wal/instance:
23:39:50.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b96c54d5bac944588ef07845f2c6874e"
23:39:50.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:50 on dist-test-slave-9jjr"
23:39:50.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.966935 6469 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
23:39:50.972 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.972602 6490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:39:50.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.973630 6469 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
23:39:50.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.974000 6469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/wal
23:39:50.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "b96c54d5bac944588ef07845f2c6874e"
23:39:50.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:50 on dist-test-slave-9jjr"
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:50.974399 6469 fs_report.cc:383] FS layout report
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/wal
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/wal
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/data/data
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:39:50.975 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:39:51.010 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.009850 6469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:39:51.011 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.011461 6469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:39:51.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.011977 6469 kserver.cc:163] Server-wide thread pool size limit: 3276
23:39:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.088697 6469 rpc_server.cc:307] RPC server started. Bound to: 127.6.78.62:46719
23:39:51.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.088821 6542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.78.62:46719 every 8 connection(s)
23:39:51.091 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.091663 6469 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/data/info.pb
23:39:51.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.097671 6543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
23:39:51.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.100627 6456 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 6469
23:39:51.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.101150 6456 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-0/wal/instance
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.104728 6456 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/wal
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/data
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/logs
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/data/info.pb
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.61:36805
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.61
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:39:51.105 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:39:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:39:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:39:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:39:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:39:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:39:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:39:51.106 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37281/jwks.json with env {}
23:39:51.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.110301 6543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } has no permanent_uuid. Determining permanent_uuid...
23:39:51.142 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.141779 6543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } has no permanent_uuid. Determining permanent_uuid...
23:39:51.144 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.144459 6494 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.78.61:36805: connect: Connection refused (error 111)
23:39:51.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.148872 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.61:36805: Network error: Client connection negotiation failed: client connection to 127.6.78.61:36805: connect: Connection refused (error 111)
23:39:51.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.198468 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } attempt: 1
23:39:51.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.202718 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.61:36805: Network error: Client connection negotiation failed: client connection to 127.6.78.61:36805: connect: Connection refused (error 111)
23:39:51.271 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.271174 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } attempt: 2
23:39:51.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.275391 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.61:36805: Network error: Client connection negotiation failed: client connection to 127.6.78.61:36805: connect: Connection refused (error 111)
23:39:51.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.366814 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } attempt: 3
23:39:51.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.370697 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.61:36805: Network error: Client connection negotiation failed: client connection to 127.6.78.61:36805: connect: Connection refused (error 111)
23:39:51.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.431695 6545 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:39:51.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.432298 6545 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:39:51.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.432796 6545 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:39:51.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.464440 6545 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
23:39:51.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.464726 6545 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:39:51.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.464926 6545 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
23:39:51.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.465116 6545 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
23:39:51.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.465312 6545 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
23:39:51.465 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.465452 6545 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37281/jwks.json
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.501274 6545 master_runner.cc:386] Master server non-default flags:
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/data
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/wal
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.61:36805
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:39:51.501 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37281/jwks.json
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/data/info.pb
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.61
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/logs
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:39:51.502 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.502517 6545 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:39:51.504 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.504036 6545 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:39:51.515 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.514627 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } attempt: 4
23:39:51.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.519400 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.61:36805: Network error: Client connection negotiation failed: client connection to 127.6.78.61:36805: connect: Connection refused (error 111)
23:39:51.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.523901 6556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:51.819 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.818805 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } attempt: 5
23:39:51.824 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.824091 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.61:36805: Network error: Client connection negotiation failed: client connection to 127.6.78.61:36805: connect: Connection refused (error 111)
23:39:52.372 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.371531 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } attempt: 6
23:39:52.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:52.378613 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.61:36805: Network error: Client connection negotiation failed: client connection to 127.6.78.61:36805: connect: Connection refused (error 111)
23:39:52.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.524073 6555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:52.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:51.525122 6559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:52.743 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:51.525629 6545 server_base.cc:1034] running on GCE node
23:39:52.747 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.746927 6545 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:39:52.750 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.749735 6545 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:39:52.751 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.751087 6545 hybrid_clock.cc:648] HybridClock initialized: now 1786405192751055 us; error 49 us; skew 500 ppm
23:39:52.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.751909 6545 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:39:52.759 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.759156 6545 webserver.cc:460] Webserver started at http://127.6.78.61:34767/ using document root <none> and password file <none>
23:39:52.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.760133 6545 fs_manager.cc:359] Metadata directory not provided
23:39:52.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.760407 6545 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:39:52.761 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.760915 6545 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
23:39:52.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.765967 6545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/data/instance:
23:39:52.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0870a8f65a2a46b59b3c0608b389e8b1"
23:39:52.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:52 on dist-test-slave-9jjr"
23:39:52.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.767233 6545 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/wal/instance:
23:39:52.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0870a8f65a2a46b59b3c0608b389e8b1"
23:39:52.767 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:52 on dist-test-slave-9jjr"
23:39:52.775 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.774808 6545 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
23:39:52.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.780763 6568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:39:52.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.781842 6545 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
23:39:52.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.782232 6545 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/wal
23:39:52.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "0870a8f65a2a46b59b3c0608b389e8b1"
23:39:52.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:52 on dist-test-slave-9jjr"
23:39:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.782608 6545 fs_report.cc:383] FS layout report
23:39:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:39:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/wal
23:39:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/wal
23:39:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/data/data
23:39:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:39:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:39:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:39:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:39:52.783 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:39:52.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:39:52.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:39:52.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:39:52.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:39:52.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:39:52.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:39:52.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.816277 6545 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:39:52.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.817871 6545 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:39:52.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.818423 6545 kserver.cc:163] Server-wide thread pool size limit: 3276
23:39:52.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.893452 6545 rpc_server.cc:307] RPC server started. Bound to: 127.6.78.61:36805
23:39:52.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.893543 6624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.78.61:36805 every 8 connection(s)
23:39:52.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.896482 6545 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/data/info.pb
23:39:52.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.901958 6625 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
23:39:52.902 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.902597 6456 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 6545
23:39:52.903 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.903040 6456 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-1/wal/instance
23:39:52.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.905452 6456 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:39:52.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:39:52.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/wal
23:39:52.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/data
23:39:52.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:39:52.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:39:52.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/logs
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/data/info.pb
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.60:44397
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.60
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:39:52.906 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37281/jwks.json with env {}
23:39:52.916 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.913893 6625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } has no permanent_uuid. Determining permanent_uuid...
23:39:52.939 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.938957 6625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } has no permanent_uuid. Determining permanent_uuid...
23:39:52.957 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:52.956605 6625 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } has no permanent_uuid. Determining permanent_uuid...
23:39:52.959 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:52.959000 6572 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:52.963 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:52.963418 6625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:53.013 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.012909 6625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 1
23:39:53.017 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.017023 6625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:53.085 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.085457 6625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 2
23:39:53.089 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.089316 6625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:53.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.180735 6625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 3
23:39:53.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.184631 6625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.234231 6627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:39:53.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.234809 6627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:39:53.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.235293 6627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:39:53.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.266011 6627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
23:39:53.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.266315 6627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:39:53.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.266499 6627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
23:39:53.266 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.266681 6627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
23:39:53.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.266881 6627 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
23:39:53.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.267020 6627 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:37281/jwks.json
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.302922 6627 master_runner.cc:386] Master server non-default flags:
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/data
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/wal
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.60:44397
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:37281/jwks.json
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/data/info.pb
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.60
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:39:53.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/logs
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:39:53.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.304350 6627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:39:53.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.306027 6627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:39:53.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.327652 6639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:53.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.328033 6625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 4
23:39:53.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.332418 6625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:53.439 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.439211 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } attempt: 7
23:39:53.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.454777 6543 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } has no permanent_uuid. Determining permanent_uuid...
23:39:53.461 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.461242 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:53.520 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.519631 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 1
23:39:53.524 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.523569 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:53.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.605216 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 2
23:39:53.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.609532 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:53.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.631891 6625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 5
23:39:53.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.637367 6625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:53.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.695170 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 3
23:39:53.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.699276 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:53.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:53.839907 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 4
23:39:53.845 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.844606 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:54.128 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.128233 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 5
23:39:54.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:54.133458 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:54.185 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.184793 6625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 6
23:39:54.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:54.191793 6625 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:54.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.327665 6641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:54.537 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:53.329430 6644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:54.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:54.537982 6642 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
23:39:54.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.538035 6627 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
23:39:54.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.542321 6627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:39:54.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.545195 6627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:39:54.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.546571 6627 hybrid_clock.cc:648] HybridClock initialized: now 1786405194546526 us; error 55 us; skew 500 ppm
23:39:54.547 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.547434 6627 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:39:54.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.554631 6627 webserver.cc:460] Webserver started at http://127.6.78.60:35699/ using document root <none> and password file <none>
23:39:54.555 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.555696 6627 fs_manager.cc:359] Metadata directory not provided
23:39:54.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.555944 6627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:39:54.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.556491 6627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
23:39:54.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.561620 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/data/instance:
23:39:54.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d83d491aa4ce473b93723eac5b31ed30"
23:39:54.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:54 on dist-test-slave-9jjr"
23:39:54.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.562846 6627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/wal/instance:
23:39:54.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d83d491aa4ce473b93723eac5b31ed30"
23:39:54.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:54 on dist-test-slave-9jjr"
23:39:54.570 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.570313 6627 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
23:39:54.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.576077 6656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:39:54.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.577097 6627 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
23:39:54.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.577461 6627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/wal
23:39:54.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d83d491aa4ce473b93723eac5b31ed30"
23:39:54.577 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:54 on dist-test-slave-9jjr"
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.577826 6627 fs_report.cc:383] FS layout report
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/wal
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/wal
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/data/data
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:39:54.578 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:39:54.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.617865 6627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:39:54.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.619467 6627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:39:54.620 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.619974 6627 kserver.cc:163] Server-wide thread pool size limit: 3276
23:39:54.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.685906 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 6
23:39:54.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:54.693308 6543 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.6.78.60:44397: Network error: Client connection negotiation failed: client connection to 127.6.78.60:44397: connect: Connection refused (error 111)
23:39:54.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.698411 6627 rpc_server.cc:307] RPC server started. Bound to: 127.6.78.60:44397
23:39:54.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.698527 6718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.78.60:44397 every 8 connection(s)
23:39:54.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.701177 6627 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/data/info.pb
23:39:54.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.703606 6456 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 6627
23:39:54.706 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.704152 6456 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/master-2/wal/instance
23:39:54.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.707269 6719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
23:39:54.731 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.729059 6719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } has no permanent_uuid. Determining permanent_uuid...
23:39:54.756 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.755721 6719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } has no permanent_uuid. Determining permanent_uuid...
23:39:54.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.768205 6719 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } has no permanent_uuid. Determining permanent_uuid...
23:39:54.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.786538 6719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30: Bootstrap starting.
23:39:54.793 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.792848 6719 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30: Neither blocks nor log segments found. Creating new log.
23:39:54.795 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.794987 6719 log.cc:824] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30: Log is configured to *not* fsync() on all Append() calls
23:39:54.799 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.799688 6719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30: No bootstrap required, opened a new log
23:39:54.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.821602 6719 raft_consensus.cc:348] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } }
23:39:54.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.822233 6719 raft_consensus.cc:374] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
23:39:54.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.822502 6719 raft_consensus.cc:729] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d83d491aa4ce473b93723eac5b31ed30, State: Initialized, Role: FOLLOWER
23:39:54.823 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.823175 6719 consensus_queue.cc:260] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } }
23:39:54.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.826261 6729 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } } }
23:39:54.827 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.827229 6729 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [sys.catalog]: This master's current role is: FOLLOWER
23:39:54.828 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.827997 6719 sys_catalog.cc:564] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [sys.catalog]: configured and running, proceeding with master startup.
23:39:54.852 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:54.852530 6741 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
23:39:54.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:54.852856 6741 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
23:39:54.876 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:54.875301 6663 tablet.cc:2369] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30: Can't schedule compaction. Clean time has not been advanced past its initial value.
23:39:54.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.849447 6627 master_runner.cc:186] Error getting master registration for 127.6.78.62:46719: OK, instance_id { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" instance_seqno: 1786405191067382 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
23:39:54.882 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:54.881795 6627 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
23:39:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.126471 6729 raft_consensus.cc:482] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
23:39:55.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.127058 6729 raft_consensus.cc:504] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } }
23:39:55.131 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.129868 6729 leader_election.cc:288] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b96c54d5bac944588ef07845f2c6874e (127.6.78.62:46719), 0870a8f65a2a46b59b3c0608b389e8b1 (127.6.78.61:36805)
23:39:55.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.252420 6625 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 7
23:39:55.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.277410 6625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1: Bootstrap starting.
23:39:55.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.284595 6625 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1: Neither blocks nor log segments found. Creating new log.
23:39:55.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.286926 6625 log.cc:824] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1: Log is configured to *not* fsync() on all Append() calls
23:39:55.292 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.292414 6625 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1: No bootstrap required, opened a new log
23:39:55.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.325189 6625 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } }
23:39:55.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.326114 6625 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
23:39:55.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.326432 6625 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0870a8f65a2a46b59b3c0608b389e8b1, State: Initialized, Role: FOLLOWER
23:39:55.328 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.327431 6625 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } }
23:39:55.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.330432 6746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } } }
23:39:55.331 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.331272 6746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [sys.catalog]: This master's current role is: FOLLOWER
23:39:55.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.332263 6625 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [sys.catalog]: configured and running, proceeding with master startup.
23:39:55.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.332463 6598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d83d491aa4ce473b93723eac5b31ed30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" is_pre_election: true
23:39:55.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.333380 6598 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d83d491aa4ce473b93723eac5b31ed30 in term 0.
23:39:55.340 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.335000 6661 leader_election.cc:302] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0870a8f65a2a46b59b3c0608b389e8b1, d83d491aa4ce473b93723eac5b31ed30; no voters:
23:39:55.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.336194 6729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 0 FOLLOWER]: Leader pre-election won for term 1
23:39:55.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.336560 6729 raft_consensus.cc:482] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
23:39:55.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.336889 6729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 0 FOLLOWER]: Advancing to term 1
23:39:55.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.351547 6729 raft_consensus.cc:504] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } }
23:39:55.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.362170 6598 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d83d491aa4ce473b93723eac5b31ed30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0870a8f65a2a46b59b3c0608b389e8b1"
23:39:55.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.362787 6598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [term 0 FOLLOWER]: Advancing to term 1
23:39:55.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.369433 6598 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d83d491aa4ce473b93723eac5b31ed30 in term 1.
23:39:55.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.370432 6661 leader_election.cc:302] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0870a8f65a2a46b59b3c0608b389e8b1, d83d491aa4ce473b93723eac5b31ed30; no voters:
23:39:55.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.361162 6729 leader_election.cc:288] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [CANDIDATE]: Term 1 election: Requested vote from peers b96c54d5bac944588ef07845f2c6874e (127.6.78.62:46719), 0870a8f65a2a46b59b3c0608b389e8b1 (127.6.78.61:36805)
23:39:55.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.375181 6729 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 1 FOLLOWER]: Leader election won for term 1
23:39:55.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.375705 6729 raft_consensus.cc:686] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [term 1 LEADER]: Becoming Leader. State: Replica: d83d491aa4ce473b93723eac5b31ed30, State: Running, Role: LEADER
23:39:55.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.376775 6729 consensus_queue.cc:237] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } }
23:39:55.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.400058 6750 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d83d491aa4ce473b93723eac5b31ed30. Latest consensus state: current_term: 1 leader_uuid: "d83d491aa4ce473b93723eac5b31ed30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } } }
23:39:55.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.403234 6750 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [sys.catalog]: This master's current role is: LEADER
23:39:55.416 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.411172 6753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
23:39:55.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.419075 6753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
23:39:55.450 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.449846 6598 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [term 1 FOLLOWER]: Refusing update from remote peer d83d491aa4ce473b93723eac5b31ed30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
23:39:55.453 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.452545 6750 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
23:39:55.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:55.471304 6762 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
23:39:55.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:55.471698 6762 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
23:39:55.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.451949 6545 master_runner.cc:186] Error getting master registration for 127.6.78.62:46719: OK, instance_id { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" instance_seqno: 1786405191067382 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
23:39:55.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.497815 6545 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
23:39:55.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.510429 6758 mvcc.cc:205] Tried to move back new op lower bound from 7317115680526073856 to 7317115680312905728. Current Snapshot: MvccSnapshot[applied={T|T < 7317115680526073856}]
23:39:55.518 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.517479 6746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d83d491aa4ce473b93723eac5b31ed30. Latest consensus state: current_term: 1 leader_uuid: "d83d491aa4ce473b93723eac5b31ed30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } } }
23:39:55.519 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.519639 6746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [sys.catalog]: This master's current role is: FOLLOWER
23:39:55.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.521802 6750 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d83d491aa4ce473b93723eac5b31ed30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } } }
23:39:55.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.522691 6750 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [sys.catalog]: This master's current role is: LEADER
23:39:55.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.524529 6753 catalog_manager.cc:1347] Generated new cluster ID: f22c984f0cee455da3f9c8ab3040fa38
23:39:55.529 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.524878 6753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
23:39:55.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.533283 6746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d83d491aa4ce473b93723eac5b31ed30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } } }
23:39:55.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.534031 6746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1 [sys.catalog]: This master's current role is: FOLLOWER
23:39:55.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.603550 6753 catalog_manager.cc:1370] Generated new certificate authority record
23:39:55.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.606823 6753 catalog_manager.cc:1504] Loading token signing keys...
23:39:55.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.645218 6753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30: Generated new TSK 0
23:39:55.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.646718 6753 catalog_manager.cc:1514] Initializing in-progress tserver states...
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.701953 6456 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/wal
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/data
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/logs
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/data/info.pb
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:39:55.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.1:0
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.78.1
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.1
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:39:55.703 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
23:39:55.727 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.726802 6543 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } attempt: 7
23:39:55.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.768940 6543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e: Bootstrap starting.
23:39:55.784 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.784159 6543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e: Neither blocks nor log segments found. Creating new log.
23:39:55.791 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.791507 6543 log.cc:824] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e: Log is configured to *not* fsync() on all Append() calls
23:39:55.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.825793 6543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e: No bootstrap required, opened a new log
23:39:55.884 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.883366 6543 raft_consensus.cc:348] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } }
23:39:55.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.893904 6543 raft_consensus.cc:374] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
23:39:55.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.894493 6543 raft_consensus.cc:729] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96c54d5bac944588ef07845f2c6874e, State: Initialized, Role: FOLLOWER
23:39:55.896 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.895810 6543 consensus_queue.cc:260] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } }
23:39:55.911 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.910691 6543 sys_catalog.cc:564] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [sys.catalog]: configured and running, proceeding with master startup.
23:39:55.914 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.911283 6515 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [term 0 FOLLOWER]: Advancing to term 1
23:39:55.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.915431 6516 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d83d491aa4ce473b93723eac5b31ed30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b96c54d5bac944588ef07845f2c6874e"
23:39:55.920 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.918068 6517 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d83d491aa4ce473b93723eac5b31ed30" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b96c54d5bac944588ef07845f2c6874e" is_pre_election: true
23:39:55.925 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.925066 6515 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [term 1 FOLLOWER]: Refusing update from remote peer d83d491aa4ce473b93723eac5b31ed30: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
23:39:55.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.926883 6729 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
23:39:55.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.935343 6772 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } } }
23:39:55.936 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.936590 6772 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [sys.catalog]: This master's current role is: FOLLOWER
23:39:55.991 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:55.990923 6778 mvcc.cc:205] Tried to move back new op lower bound from 7317115680526073856 to 7317115680312905728. Current Snapshot: MvccSnapshot[applied={T|T < 7317115680526073856}]
23:39:56.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.016018 6772 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d83d491aa4ce473b93723eac5b31ed30. Latest consensus state: current_term: 1 leader_uuid: "d83d491aa4ce473b93723eac5b31ed30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } } }
23:39:56.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.016796 6772 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [sys.catalog]: This master's current role is: FOLLOWER
23:39:56.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.017987 6775 sys_catalog.cc:455] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d83d491aa4ce473b93723eac5b31ed30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } } }
23:39:56.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.018819 6775 sys_catalog.cc:458] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e [sys.catalog]: This master's current role is: FOLLOWER
23:39:56.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.033583 6790 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
23:39:56.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:56.309806 6768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:39:56.310 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:56.310441 6768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:39:56.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:56.311036 6768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:39:56.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:56.352653 6768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:39:56.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:56.354616 6768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.78.1
23:39:56.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.366606 6729 sys_catalog.cc:455] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d83d491aa4ce473b93723eac5b31ed30" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b96c54d5bac944588ef07845f2c6874e" member_type: VOTER last_known_addr { host: "127.6.78.62" port: 46719 } } peers { permanent_uuid: "0870a8f65a2a46b59b3c0608b389e8b1" member_type: VOTER last_known_addr { host: "127.6.78.61" port: 36805 } } peers { permanent_uuid: "d83d491aa4ce473b93723eac5b31ed30" member_type: VOTER last_known_addr { host: "127.6.78.60" port: 44397 } } }
23:39:56.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.367720 6729 sys_catalog.cc:458] T 00000000000000000000000000000000 P d83d491aa4ce473b93723eac5b31ed30 [sys.catalog]: This master's current role is: LEADER
23:39:56.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.403808 6768 tablet_server_runner.cc:78] Tablet server non-default flags:
23:39:56.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:39:56.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:39:56.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:39:56.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:39:56.404 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/data
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/wal
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.1:0
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/data/info.pb
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.1
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.78.1
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/logs
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:39:56.405 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.405393 6768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:39:56.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.407446 6768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:39:56.434 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:56.434172 6799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:56.479 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.479398 6762 catalog_manager.cc:1259] Loaded cluster ID: f22c984f0cee455da3f9c8ab3040fa38
23:39:56.480 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.480008 6762 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1: loading cluster ID for follower catalog manager: success
23:39:56.503 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.502539 6762 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1: acquiring CA information for follower catalog manager: success
23:39:56.511 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:56.510540 6762 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0870a8f65a2a46b59b3c0608b389e8b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
23:39:57.062 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:57.062433 6790 catalog_manager.cc:1259] Loaded cluster ID: f22c984f0cee455da3f9c8ab3040fa38
23:39:57.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:57.070243 6790 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e: loading cluster ID for follower catalog manager: success
23:39:57.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:57.094529 6790 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e: acquiring CA information for follower catalog manager: success
23:39:57.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:57.100587 6790 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b96c54d5bac944588ef07845f2c6874e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
23:39:58.270 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:56.439589 6803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:58.272 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:56.434455 6801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:58.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:58.275544 6802 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1835 milliseconds
23:39:58.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.275728 6768 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
23:39:58.279 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.277290 6768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:39:58.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.284399 6768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:39:58.286 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.285983 6768 hybrid_clock.cc:648] HybridClock initialized: now 1786405198285935 us; error 49 us; skew 500 ppm
23:39:58.287 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.287163 6768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:39:58.300 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.300280 6768 webserver.cc:460] Webserver started at http://127.6.78.1:44029/ using document root <none> and password file <none>
23:39:58.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.301558 6768 fs_manager.cc:359] Metadata directory not provided
23:39:58.302 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.301839 6768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:39:58.303 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.302886 6768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
23:39:58.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.310967 6768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/data/instance:
23:39:58.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "27800651fe5b47a6b3278dde45f6d89c"
23:39:58.311 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:58 on dist-test-slave-9jjr"
23:39:58.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.312633 6768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/wal/instance:
23:39:58.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "27800651fe5b47a6b3278dde45f6d89c"
23:39:58.314 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:58 on dist-test-slave-9jjr"
23:39:58.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.324791 6768 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
23:39:58.332 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.332536 6812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:39:58.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.333973 6768 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
23:39:58.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.334509 6768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/wal
23:39:58.334 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "27800651fe5b47a6b3278dde45f6d89c"
23:39:58.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:39:58 on dist-test-slave-9jjr"
23:39:58.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.335145 6768 fs_report.cc:383] FS layout report
23:39:58.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:39:58.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/wal
23:39:58.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/wal
23:39:58.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/data/data
23:39:58.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:39:58.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:39:58.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:39:58.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:39:58.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:39:58.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:39:58.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:39:58.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:39:58.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:39:58.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:39:58.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:39:58.417 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.416877 6768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:39:58.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.419113 6768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:39:58.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.419857 6768 kserver.cc:163] Server-wide thread pool size limit: 3276
23:39:58.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.423591 6768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
23:39:58.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.429877 6768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
23:39:58.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.430274 6768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
23:39:58.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.430696 6768 ts_tablet_manager.cc:610] Registered 0 tablets
23:39:58.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.431010 6768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
23:39:58.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.608229 6768 rpc_server.cc:307] RPC server started. Bound to: 127.6.78.1:36563
23:39:58.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.608340 6942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.78.1:36563 every 8 connection(s)
23:39:58.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.611064 6768 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/data/info.pb
23:39:58.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.618901 6456 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 6768
23:39:58.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.619385 6456 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-0/wal/instance
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.631898 6456 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/wal
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/data
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:39:58.632 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/logs
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/data/info.pb
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.2:0
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.78.2
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.2
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:39:58.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:39:58.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:39:58.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
23:39:58.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.674276 6943 heartbeater.cc:347] Connected to a master server at 127.6.78.61:36805
23:39:58.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.674813 6943 heartbeater.cc:464] Registering TS with master...
23:39:58.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.675873 6944 heartbeater.cc:347] Connected to a master server at 127.6.78.60:44397
23:39:58.676 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.676144 6943 heartbeater.cc:511] Master 127.6.78.61:36805 requested a full tablet report, sending...
23:39:58.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.676237 6944 heartbeater.cc:464] Registering TS with master...
23:39:58.678 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.677865 6944 heartbeater.cc:511] Master 127.6.78.60:44397 requested a full tablet report, sending...
23:39:58.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.680209 6588 ts_manager.cc:194] Registered new tserver with Master: 27800651fe5b47a6b3278dde45f6d89c (127.6.78.1:36563)
23:39:58.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.680646 6675 ts_manager.cc:194] Registered new tserver with Master: 27800651fe5b47a6b3278dde45f6d89c (127.6.78.1:36563)
23:39:58.683 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.683315 6675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.78.1:38831
23:39:58.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.684861 6945 heartbeater.cc:347] Connected to a master server at 127.6.78.62:46719
23:39:58.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.685185 6945 heartbeater.cc:464] Registering TS with master...
23:39:58.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.686021 6945 heartbeater.cc:511] Master 127.6.78.62:46719 requested a full tablet report, sending...
23:39:58.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:58.688818 6507 ts_manager.cc:194] Registered new tserver with Master: 27800651fe5b47a6b3278dde45f6d89c (127.6.78.1:36563)
23:39:59.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:59.037318 6954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:39:59.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:59.037786 6954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:39:59.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:59.038314 6954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:39:59.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:59.069948 6954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:39:59.071 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:59.070943 6954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.78.2
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:59.123557 6954 tablet_server_runner.cc:78] Tablet server non-default flags:
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/data
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/wal
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.2:0
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/data/info.pb
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.2
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:39:59.124 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.78.2
23:39:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/logs
23:39:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:39:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:39:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:39:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
23:39:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:39:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:39:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:39:59.125 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:39:59.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:39:59.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:39:59.126 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:59.125149 6954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:39:59.127 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:59.127205 6954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:39:59.149 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:59.148665 6963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:39:59.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:39:59.688275 6944 heartbeater.cc:503] Master 127.6.78.60:44397 was elected leader, sending a full tablet report...
23:40:00.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:59.150954 6962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:00.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:39:59.148842 6965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:00.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:00.560115 6964 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1410 milliseconds
23:40:00.560 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.560215 6954 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
23:40:00.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.561570 6954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:40:00.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.564586 6954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:40:00.566 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.566023 6954 hybrid_clock.cc:648] HybridClock initialized: now 1786405200565954 us; error 79 us; skew 500 ppm
23:40:00.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.566887 6954 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:40:00.574 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.573954 6954 webserver.cc:460] Webserver started at http://127.6.78.2:33111/ using document root <none> and password file <none>
23:40:00.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.574882 6954 fs_manager.cc:359] Metadata directory not provided
23:40:00.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.575141 6954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:40:00.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.575637 6954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
23:40:00.580 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.580477 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/data/instance:
23:40:00.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2d8d0db0a41144d0b2d23cb5594fe650"
23:40:00.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:00 on dist-test-slave-9jjr"
23:40:00.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.581681 6954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/wal/instance:
23:40:00.581 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2d8d0db0a41144d0b2d23cb5594fe650"
23:40:00.582 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:00 on dist-test-slave-9jjr"
23:40:00.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.588938 6954 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
23:40:00.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.594754 6975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:40:00.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.596001 6954 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
23:40:00.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.596297 6954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/wal
23:40:00.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "2d8d0db0a41144d0b2d23cb5594fe650"
23:40:00.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:00 on dist-test-slave-9jjr"
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.596622 6954 fs_report.cc:383] FS layout report
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/wal
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/wal
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/data/data
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:40:00.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:40:00.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.641170 6954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:40:00.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.642730 6954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:40:00.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.643204 6954 kserver.cc:163] Server-wide thread pool size limit: 3276
23:40:00.646 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.645812 6954 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
23:40:00.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.650092 6954 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
23:40:00.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.650311 6954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
23:40:00.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.650537 6954 ts_tablet_manager.cc:610] Registered 0 tablets
23:40:00.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.650668 6954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
23:40:00.808 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.808554 6954 rpc_server.cc:307] RPC server started. Bound to: 127.6.78.2:45133
23:40:00.809 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.808670 7094 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.78.2:45133 every 8 connection(s)
23:40:00.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.811193 6954 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/data/info.pb
23:40:00.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.813750 6456 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 6954
23:40:00.818 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.814734 6456 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-1/wal/instance
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.833544 6456 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/wal
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/data
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/logs
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/data/info.pb
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) tserver
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.3:0
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.78.3
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.3
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:00.834 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin with env {}
23:40:00.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.853226 7095 heartbeater.cc:347] Connected to a master server at 127.6.78.61:36805
23:40:00.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.853560 7097 heartbeater.cc:347] Connected to a master server at 127.6.78.62:46719
23:40:00.853 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.853783 7095 heartbeater.cc:464] Registering TS with master...
23:40:00.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.854040 7097 heartbeater.cc:464] Registering TS with master...
23:40:00.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.854359 7096 heartbeater.cc:347] Connected to a master server at 127.6.78.60:44397
23:40:00.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.854658 7096 heartbeater.cc:464] Registering TS with master...
23:40:00.855 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.855409 7095 heartbeater.cc:511] Master 127.6.78.61:36805 requested a full tablet report, sending...
23:40:00.856 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.855512 7097 heartbeater.cc:511] Master 127.6.78.62:46719 requested a full tablet report, sending...
23:40:00.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.855466 7096 heartbeater.cc:511] Master 127.6.78.60:44397 requested a full tablet report, sending...
23:40:00.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.858757 6588 ts_manager.cc:194] Registered new tserver with Master: 2d8d0db0a41144d0b2d23cb5594fe650 (127.6.78.2:45133)
23:40:00.859 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.859128 6507 ts_manager.cc:194] Registered new tserver with Master: 2d8d0db0a41144d0b2d23cb5594fe650 (127.6.78.2:45133)
23:40:00.860 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.860502 6675 ts_manager.cc:194] Registered new tserver with Master: 2d8d0db0a41144d0b2d23cb5594fe650 (127.6.78.2:45133)
23:40:00.862 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:00.862277 6675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.78.2:45489
23:40:01.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:01.163173 7106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:40:01.163 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:01.163640 7106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:40:01.164 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:01.164170 7106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:40:01.195 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:01.195535 7106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:40:01.196 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:01.196352 7106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.78.3
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:01.231860 7106 tablet_server_runner.cc:78] Tablet server non-default flags:
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.6.78.20:41715
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/data
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/wal
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.6.78.3:0
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/data/info.pb
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.6.78.3
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tserver_master_addrs=127.6.78.62:46719,127.6.78.61:36805,127.6.78.60:44397
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --local_ip_for_outbound_sockets=127.6.78.3
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/logs
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Tablet server version:
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:40:01.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:40:01.233 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:01.233162 7106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:40:01.235 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:01.234831 7106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:40:01.255 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:01.254487 7116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:01.868 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:01.866461 7096 heartbeater.cc:503] Master 127.6.78.60:44397 was elected leader, sending a full tablet report...
23:40:02.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:01.254495 7113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:02.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:01.254531 7114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:02.596733 7115 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1344 milliseconds
23:40:02.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.596879 7106 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
23:40:02.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.601011 7106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:40:02.604 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.603883 7106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:40:02.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.605476 7106 hybrid_clock.cc:648] HybridClock initialized: now 1786405202605395 us; error 90 us; skew 500 ppm
23:40:02.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.606379 7106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:40:02.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.614403 7106 webserver.cc:460] Webserver started at http://127.6.78.3:43813/ using document root <none> and password file <none>
23:40:02.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.615398 7106 fs_manager.cc:359] Metadata directory not provided
23:40:02.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.615600 7106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:40:02.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.615996 7106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
23:40:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.621269 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/data/instance:
23:40:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "83cbafe6cd284d1dab7fa7235777af82"
23:40:02.621 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:02 on dist-test-slave-9jjr"
23:40:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.622537 7106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/wal/instance:
23:40:02.622 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "83cbafe6cd284d1dab7fa7235777af82"
23:40:02.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:02 on dist-test-slave-9jjr"
23:40:02.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.630371 7106 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
23:40:02.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.636220 7125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:40:02.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.637387 7106 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
23:40:02.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.637676 7106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/wal
23:40:02.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "83cbafe6cd284d1dab7fa7235777af82"
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:02 on dist-test-slave-9jjr"
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.637971 7106 fs_report.cc:383] FS layout report
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/wal
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/wal
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/data/data
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:40:02.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:40:02.704 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.704337 7106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:40:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.706352 7106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:40:02.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.706959 7106 kserver.cc:163] Server-wide thread pool size limit: 3276
23:40:02.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.711364 7106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
23:40:02.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.716248 7106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
23:40:02.716 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.716506 7106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
23:40:02.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.716867 7106 ts_tablet_manager.cc:610] Registered 0 tablets
23:40:02.717 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.717056 7106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
23:40:02.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.869347 7106 rpc_server.cc:307] RPC server started. Bound to: 127.6.78.3:41427
23:40:02.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.869549 7249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.6.78.3:41427 every 8 connection(s)
23:40:02.873 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.872879 7106 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/data/info.pb
23:40:02.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.874316 6456 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 7106
23:40:02.878 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.874797 6456 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster5824436419617816997/ts-2/wal/instance
23:40:02.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.936564 7250 heartbeater.cc:347] Connected to a master server at 127.6.78.61:36805
23:40:02.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.937129 7250 heartbeater.cc:464] Registering TS with master...
23:40:02.941 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.938436 7250 heartbeater.cc:511] Master 127.6.78.61:36805 requested a full tablet report, sending...
23:40:02.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.938949 7251 heartbeater.cc:347] Connected to a master server at 127.6.78.60:44397
23:40:02.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.939343 7251 heartbeater.cc:464] Registering TS with master...
23:40:02.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.940222 7251 heartbeater.cc:511] Master 127.6.78.60:44397 requested a full tablet report, sending...
23:40:02.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.942461 7253 heartbeater.cc:347] Connected to a master server at 127.6.78.62:46719
23:40:02.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.942673 6675 ts_manager.cc:194] Registered new tserver with Master: 83cbafe6cd284d1dab7fa7235777af82 (127.6.78.3:41427)
23:40:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.942829 7253 heartbeater.cc:464] Registering TS with master...
23:40:02.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.943674 7253 heartbeater.cc:511] Master 127.6.78.62:46719 requested a full tablet report, sending...
23:40:02.946 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.944782 6675 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.6.78.3:39881
23:40:02.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.946369 6507 ts_manager.cc:194] Registered new tserver with Master: 83cbafe6cd284d1dab7fa7235777af82 (127.6.78.3:41427)
23:40:02.947 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.946473 6588 ts_manager.cc:194] Registered new tserver with Master: 83cbafe6cd284d1dab7fa7235777af82 (127.6.78.3:41427)
23:40:02.956 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:02.956138 6456 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
23:40:02.967 [DEBUG - main] (MiniKuduCluster.java:195) Response:
23:40:02.967 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_masters {
}
23:40:02.970 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.6.78.62"
port: 46719
}
bound_http_address {
host: "127.6.78.62"
port: 45937
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.6.78.61"
port: 36805
}
bound_http_address {
host: "127.6.78.61"
port: 34767
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.6.78.60"
port: 44397
}
bound_http_address {
host: "127.6.78.60"
port: 35699
}
}
}
23:40:02.970 [DEBUG - main] (MiniKuduCluster.java:185) Request: get_tservers {
}
23:40:02.971 [DEBUG - main] (MiniKuduCluster.java:195) Response: get_tservers {
tservers {
id {
type: TSERVER
index: 0
}
bound_rpc_address {
host: "127.6.78.1"
port: 36563
}
bound_http_address {
host: "127.6.78.1"
port: 44029
}
}
tservers {
id {
type: TSERVER
index: 1
}
bound_rpc_address {
host: "127.6.78.2"
port: 45133
}
bound_http_address {
host: "127.6.78.2"
port: 33111
}
}
tservers {
id {
type: TSERVER
index: 2
}
bound_rpc_address {
host: "127.6.78.3"
port: 41427
}
bound_http_address {
host: "127.6.78.3"
port: 43813
}
}
}
23:40:02.971 [INFO - main] (KuduTestHarness.java:157) Creating a new Kudu client...
23:40:02.972 [DEBUG - main] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster8812805469506080277/krb5kdc/krb5cc
23:40:02.973 [DEBUG - main] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
23:40:02.974 [INFO - Time-limited test] (TempDirUtils.java:71) Using the temp directory defined by TEST_TMPDIR: /tmp/dist-test-tasksngkcS/test-tmp
23:40:02.974 [INFO - Time-limited test] (KuduBinaryLocator.java:83) Using Kudu binary directory specified by system property 'kuduBinDir': ../../../build/tsan/bin
23:40:02.975 [INFO - Time-limited test] (MiniKuduCluster.java:217) Starting process: [/tmp/dist-test-tasksngkcS/java/build/dist-test/../../../build/tsan/bin/kudu, test, mini_cluster, --serialization=pb]
23:40:02.976 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: create_cluster {
num_masters: 3
num_tservers: 0
enable_kerberos: true
cluster_root: "/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588"
extra_master_flags: "--authn_token_validity_seconds=10"
hms_mode: NONE
mini_kdc_options {
ticket_lifetime: "10s"
renew_lifetime: "20s"
}
principal: "kudu"
mini_oidc_options {
}
}
23:40:03.335 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:03.336 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_cluster {
}
23:40:03.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:03.340775 7265 mini_oidc.cc:152] Starting JWKS server
23:40:03.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:03.344385 7265 webserver.cc:460] Webserver started at http://127.0.0.1:34253/ using document root <none> and password file <none>
23:40:03.344 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:03.344799 7265 mini_oidc.cc:189] Starting OIDC Discovery server
23:40:03.346 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:03.346629 7265 webserver.cc:460] Webserver started at http://127.0.0.1:34587/ using document root <none> and password file <none>
23:40:03.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Loading random data
23:40:03.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Initializing database '/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/principal' for realm 'KRBTEST.COM',
23:40:03.418 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master key name 'K/M@KRBTEST.COM'
23:40:03.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:03 dist-test-slave-9jjr krb5kdc[7277](info): setting up network...
23:40:03.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
23:40:03.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:03 dist-test-slave-9jjr krb5kdc[7277](info): set up 2 sockets
23:40:03.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:03 dist-test-slave-9jjr krb5kdc[7277](info): commencing operation
23:40:03.438 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) krb5kdc: starting...
23:40:03.952 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:03.951391 7251 heartbeater.cc:503] Master 127.6.78.60:44397 was elected leader, sending a full tablet report...
23:40:04.542 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:04.542523 7265 mini_kdc.cc:121] Time spent starting KDC: real 1.170s user 0.007s sys 0.021s
23:40:04.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
23:40:04.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
23:40:04.572 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-admin@KRBTEST.COM" created.
23:40:04.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
23:40:04.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
23:40:04.603 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "test-user@KRBTEST.COM" created.
23:40:04.623 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
23:40:04.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
23:40:04.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "joe-interloper@KRBTEST.COM" created.
23:40:04.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal slave/admin@KRBTEST.COM with password.
23:40:04.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/test-user.keytab.
23:40:04.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/test-user.keytab.
23:40:04.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:04 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:04.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
23:40:04.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-10T23:40:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
23:40:04.711 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-10T23:40:04Z Disabled control of system clock
23:40:04.742 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.7.24.126@KRBTEST.COM; defaulting to no policy
23:40:04.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
23:40:04.744 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.7.24.126@KRBTEST.COM" created.
23:40:04.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
23:40:04.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.7.24.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.126.keytab.
23:40:04.768 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.7.24.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.126.keytab.
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:04.770726 7265 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/logs
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data/info.pb
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.126:39219
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.126
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:40:04.771 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.126.keytab
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.126
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:04.772 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
23:40:05.100 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.100467 7292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:40:05.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.101033 7292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:40:05.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.101657 7292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:40:05.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.134974 7292 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
23:40:05.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.135223 7292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
23:40:05.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.135576 7292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:40:05.135 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.135785 7292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
23:40:05.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.136062 7292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
23:40:05.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.136286 7292 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
23:40:05.136 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.136431 7292 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34253/jwks.json
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:05.181353 7292 master_runner.cc:386] Master server non-default flags:
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.126.keytab
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.126
23:40:05.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.126:39219
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data/info.pb
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.126
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/logs
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:40:05.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:05.182566 7292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:40:05.183 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:05.183539 7292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:40:05.203 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.203531 7298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:06.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.203526 7299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:06.454 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:05.204131 7301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:06.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:06.455334 7300 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
23:40:06.455 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.455456 7292 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
23:40:06.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.459251 7292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:40:06.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.462237 7292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:40:06.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.463636 7292 hybrid_clock.cc:648] HybridClock initialized: now 1786405206463651 us; error 86 us; skew 500 ppm
23:40:06.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.464506 7292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:40:06.470 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:06 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405206, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:06.471 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.470788 7292 init.cc:377] Logged in from keytab as kudu/127.7.24.126@KRBTEST.COM (short username kudu)
23:40:06.476 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.476205 7292 webserver.cc:460] Webserver started at http://127.7.24.126:41995/ using document root <none> and password file <none>
23:40:06.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.477092 7292 fs_manager.cc:359] Metadata directory not provided
23:40:06.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.477275 7292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:40:06.477 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.477663 7292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
23:40:06.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.482704 7292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data/instance:
23:40:06.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d2b57669bfc643f3a807faff97cbc52e"
23:40:06.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:06 on dist-test-slave-9jjr"
23:40:06.483 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.483769 7292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal/instance:
23:40:06.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d2b57669bfc643f3a807faff97cbc52e"
23:40:06.484 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:06 on dist-test-slave-9jjr"
23:40:06.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.491130 7292 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
23:40:06.497 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.497196 7312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:40:06.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.498663 7292 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.499058 7292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d2b57669bfc643f3a807faff97cbc52e"
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:06 on dist-test-slave-9jjr"
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.499384 7292 fs_report.cc:383] FS layout report
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data/data
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:40:06.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:40:06.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:40:06.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:40:06.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:40:06.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:40:06.500 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:40:06.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.539292 7292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:40:06.545 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.545598 7292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:40:06.546 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.546037 7292 kserver.cc:163] Server-wide thread pool size limit: 3276
23:40:06.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.627792 7292 rpc_server.cc:307] RPC server started. Bound to: 127.7.24.126:39219
23:40:06.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.627832 7365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.24.126:39219 every 8 connection(s)
23:40:06.630 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.630661 7292 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data/info.pb
23:40:06.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.636382 7366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
23:40:06.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.638734 7265 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 7292
23:40:06.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.639246 7265 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal/instance
23:40:06.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.648572 7366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } has no permanent_uuid. Determining permanent_uuid...
23:40:06.664 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.7.24.125@KRBTEST.COM; defaulting to no policy
23:40:06.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
23:40:06.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.7.24.125@KRBTEST.COM" created.
23:40:06.682 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:06 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405206, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.126@KRBTEST.COM for kudu/127.7.24.126@KRBTEST.COM
23:40:06.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
23:40:06.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.7.24.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.125.keytab.
23:40:06.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.7.24.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.125.keytab.
23:40:06.698 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.698524 7366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } has no permanent_uuid. Determining permanent_uuid...
23:40:06.699 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.699412 7265 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/logs
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data/info.pb
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.125:35285
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.125
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.125.keytab
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.125
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:06.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:06.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:06.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:06.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:06.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:06.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:06.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
23:40:06.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:06.702308 7316 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.24.125:35285: connect: Connection refused (error 111)
23:40:06.710 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:06.709967 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.125:35285: Network error: Client connection negotiation failed: client connection to 127.7.24.125:35285: connect: Connection refused (error 111)
23:40:06.760 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.759641 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } attempt: 1
23:40:06.764 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:06.763614 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.125:35285: Network error: Client connection negotiation failed: client connection to 127.7.24.125:35285: connect: Connection refused (error 111)
23:40:06.832 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.832089 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } attempt: 2
23:40:06.836 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:06.836285 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.125:35285: Network error: Client connection negotiation failed: client connection to 127.7.24.125:35285: connect: Connection refused (error 111)
23:40:06.928 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:06.927740 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } attempt: 3
23:40:06.932 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:06.931985 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.125:35285: Network error: Client connection negotiation failed: client connection to 127.7.24.125:35285: connect: Connection refused (error 111)
23:40:07.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.033361 7375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:40:07.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.033921 7375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:40:07.034 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.034655 7375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:40:07.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.066866 7375 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
23:40:07.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.067106 7375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
23:40:07.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.067379 7375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:40:07.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.067582 7375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
23:40:07.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.067888 7375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
23:40:07.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.068204 7375 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
23:40:07.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.068439 7375 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34253/jwks.json
23:40:07.075 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:07.075385 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } attempt: 4
23:40:07.079 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.079146 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.125:35285: Network error: Client connection negotiation failed: client connection to 127.7.24.125:35285: connect: Connection refused (error 111)
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:07.109189 7375 master_runner.cc:386] Master server non-default flags:
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.125.keytab
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.125
23:40:07.109 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.125:35285
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data/info.pb
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.125
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/logs
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:40:07.110 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:40:07.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:40:07.111 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:07.110786 7375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:40:07.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:07.112016 7375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:40:07.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.131773 7384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:07.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:07.378551 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } attempt: 5
23:40:07.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.382356 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.125:35285: Network error: Client connection negotiation failed: client connection to 127.7.24.125:35285: connect: Connection refused (error 111)
23:40:07.930 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:07.929817 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } attempt: 6
23:40:07.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.936916 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.125:35285: Network error: Client connection negotiation failed: client connection to 127.7.24.125:35285: connect: Connection refused (error 111)
23:40:08.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.132397 7382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:08.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:07.134490 7375 server_base.cc:1034] running on GCE node
23:40:08.367 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:07.134681 7381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:08.371 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.371443 7375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:40:08.374 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.374222 7375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:40:08.375 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.375560 7375 hybrid_clock.cc:648] HybridClock initialized: now 1786405208375538 us; error 44 us; skew 500 ppm
23:40:08.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.376339 7375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:40:08.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:08 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405208, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:08.382 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.382555 7375 init.cc:377] Logged in from keytab as kudu/127.7.24.125@KRBTEST.COM (short username kudu)
23:40:08.387 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.387507 7375 webserver.cc:460] Webserver started at http://127.7.24.125:45509/ using document root <none> and password file <none>
23:40:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.388594 7375 fs_manager.cc:359] Metadata directory not provided
23:40:08.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.388784 7375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:40:08.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.389199 7375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
23:40:08.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.394306 7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data/instance:
23:40:08.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "faca6f033bfd41d4a65bce873a0bfafa"
23:40:08.394 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:08 on dist-test-slave-9jjr"
23:40:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.395576 7375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal/instance:
23:40:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "faca6f033bfd41d4a65bce873a0bfafa"
23:40:08.395 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:08 on dist-test-slave-9jjr"
23:40:08.403 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.403267 7375 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
23:40:08.409 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.409336 7397 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:40:08.410 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.410496 7375 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.410851 7375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "faca6f033bfd41d4a65bce873a0bfafa"
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:08 on dist-test-slave-9jjr"
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.411212 7375 fs_report.cc:383] FS layout report
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data/data
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:40:08.411 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:40:08.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:40:08.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:40:08.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:40:08.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:40:08.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:40:08.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:40:08.456 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.455798 7375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:40:08.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.462232 7375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:40:08.463 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.462749 7375 kserver.cc:163] Server-wide thread pool size limit: 3276
23:40:08.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.540491 7375 rpc_server.cc:307] RPC server started. Bound to: 127.7.24.125:35285
23:40:08.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.540572 7452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.24.125:35285 every 8 connection(s)
23:40:08.543 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.543398 7375 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data/info.pb
23:40:08.548 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.548602 7453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
23:40:08.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.552142 7265 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 7375
23:40:08.552 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.552508 7265 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal/instance
23:40:08.563 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.560361 7453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } has no permanent_uuid. Determining permanent_uuid...
23:40:08.573 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) WARNING: no policy specified for kudu/127.7.24.124@KRBTEST.COM; defaulting to no policy
23:40:08.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
23:40:08.576 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Principal "kudu/127.7.24.124@KRBTEST.COM" created.
23:40:08.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:08 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405208, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.125@KRBTEST.COM for kudu/127.7.24.126@KRBTEST.COM
23:40:08.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.601449 7453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } has no permanent_uuid. Determining permanent_uuid...
23:40:08.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Authenticating as principal test-admin/admin@KRBTEST.COM with password.
23:40:08.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.7.24.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.124.keytab.
23:40:08.605 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Entry for principal kudu/127.7.24.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.124.keytab.
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.608942 7265 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/logs
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data/info.pb
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:08.609 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.124:41745
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.124
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.124.keytab
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.124
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
23:40:08.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
23:40:08.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:08.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:08.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:08.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:08.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:08.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:08.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:08.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:08.611 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
23:40:08.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:08 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405208, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.125@KRBTEST.COM for kudu/127.7.24.125@KRBTEST.COM
23:40:08.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.639755 7453 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } has no permanent_uuid. Determining permanent_uuid...
23:40:08.642 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:08.642307 7398 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:08.647 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:08.646831 7453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:08.697 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.696377 7453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 1
23:40:08.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:08.700893 7453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:08.769 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.769327 7453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 2
23:40:08.774 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:08.773689 7453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:08.865 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.865165 7453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 3
23:40:08.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:08.868961 7453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:08.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:08.963797 7461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:40:08.964 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:08.964359 7461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:40:08.965 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:08.964931 7461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:40:08.998 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:08.997546 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } attempt: 7
23:40:08.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:08.999106 7461 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
23:40:08.999 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:08.999639 7461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
23:40:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.000129 7461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:40:09.000 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.000443 7461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
23:40:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.000870 7461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
23:40:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.001118 7461 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
23:40:09.001 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.001264 7461 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34253/jwks.json
23:40:09.012 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.012347 7453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 4
23:40:09.014 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:09 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405206, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.126@KRBTEST.COM for kudu/127.7.24.125@KRBTEST.COM
23:40:09.016 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.016676 7453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:09.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.029035 7366 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } has no permanent_uuid. Determining permanent_uuid...
23:40:09.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.034991 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:09.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.052978 7461 master_runner.cc:386] Master server non-default flags:
23:40:09.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:09.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:09.053 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.124.keytab
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.124
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.124:41745
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data/info.pb
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.124
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=0
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/logs
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
23:40:09.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:40:09.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:40:09.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:40:09.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:40:09.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:40:09.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:40:09.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.054524 7461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:40:09.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.055719 7461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:40:09.076 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.075945 7472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:09.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.093449 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 1
23:40:09.097 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.097579 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:09.179 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.178953 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 2
23:40:09.184 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.183831 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:09.269 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.269251 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 3
23:40:09.274 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.274065 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:09.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.316150 7453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 5
23:40:09.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.321002 7453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:09.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.414520 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 4
23:40:09.419 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.418946 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:09.702 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.702366 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 5
23:40:09.707 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.707237 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:09.869 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.868455 7453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 6
23:40:09.881 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.881036 7453 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:10.260 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.259667 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 6
23:40:10.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:10.267058 7366 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.7.24.124:41745: Network error: Client connection negotiation failed: client connection to 127.7.24.124:41745: connect: Connection refused (error 111)
23:40:10.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.077337 7471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:10.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:09.076117 7475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:10.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:09.078019 7461 server_base.cc:1034] running on GCE node
23:40:10.338 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.338114 7461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:40:10.341 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.341418 7461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:40:10.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.342851 7461 hybrid_clock.cc:648] HybridClock initialized: now 1786405210342794 us; error 80 us; skew 500 ppm
23:40:10.343 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.343698 7461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:40:10.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:10 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405210, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:10.349 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.349694 7461 init.cc:377] Logged in from keytab as kudu/127.7.24.124@KRBTEST.COM (short username kudu)
23:40:10.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.355165 7461 webserver.cc:460] Webserver started at http://127.7.24.124:32803/ using document root <none> and password file <none>
23:40:10.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.356117 7461 fs_manager.cc:359] Metadata directory not provided
23:40:10.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.356320 7461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:40:10.356 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.356721 7461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
23:40:10.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.361910 7461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data/instance:
23:40:10.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d6ecad18d3e94677b751c668ead6c7ec"
23:40:10.362 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:10 on dist-test-slave-9jjr"
23:40:10.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.363088 7461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal/instance:
23:40:10.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d6ecad18d3e94677b751c668ead6c7ec"
23:40:10.363 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:10 on dist-test-slave-9jjr"
23:40:10.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.370501 7461 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
23:40:10.376 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.376668 7490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:40:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.377810 7461 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
23:40:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.378131 7461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal
23:40:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d6ecad18d3e94677b751c668ead6c7ec"
23:40:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:10 on dist-test-slave-9jjr"
23:40:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.378489 7461 fs_report.cc:383] FS layout report
23:40:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:40:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal
23:40:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal
23:40:10.378 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data/data
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:40:10.379 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:40:10.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.429728 7461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:40:10.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.437033 7461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:40:10.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.437476 7461 kserver.cc:163] Server-wide thread pool size limit: 3276
23:40:10.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.513329 7461 rpc_server.cc:307] RPC server started. Bound to: 127.7.24.124:41745
23:40:10.513 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.513414 7547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.24.124:41745 every 8 connection(s)
23:40:10.516 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.516259 7461 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data/info.pb
23:40:10.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.521839 7548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
23:40:10.522 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.522445 7265 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 7461
23:40:10.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.522995 7265 external_mini_cluster.cc:1402] Reading /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal/instance
23:40:10.539 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.536975 7548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } has no permanent_uuid. Determining permanent_uuid...
23:40:10.561 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:10 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405210, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.124@KRBTEST.COM for kudu/127.7.24.126@KRBTEST.COM
23:40:10.564 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:10 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.24.124@KRBTEST.COM
23:40:10.575 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.575271 7548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } has no permanent_uuid. Determining permanent_uuid...
23:40:10.589 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:10 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405210, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.124@KRBTEST.COM for kudu/127.7.24.125@KRBTEST.COM
23:40:10.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:10 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.24.125@KRBTEST.COM
23:40:10.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.601472 7548 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } has no permanent_uuid. Determining permanent_uuid...
23:40:10.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:10 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405210, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.124@KRBTEST.COM for kudu/127.7.24.124@KRBTEST.COM
23:40:10.625 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:10 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405204, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.24.126@KRBTEST.COM
23:40:10.641 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.641274 7548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: Bootstrap starting.
23:40:10.645 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.644732 7265 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
23:40:10.648 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.648387 7548 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: Neither blocks nor log segments found. Creating new log.
23:40:10.651 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.650929 7548 log.cc:824] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: Log is configured to *not* fsync() on all Append() calls
23:40:10.652 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:10.653 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_kdc_env_vars {
}
23:40:10.654 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_kdc_env_vars {
env_vars {
key: "KRB5CCNAME"
value: "/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc"
}
env_vars {
key: "KRB5_CONFIG"
value: "/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5.conf"
}
env_vars {
key: "KRB5_KDC_PROFILE"
value: "/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kdc.conf"
}
env_vars {
key: "KUDU_ENABLE_KRB5_REALM_FIX"
value: "yes"
}
}
23:40:10.655 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_masters {
}
23:40:10.656 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response: get_masters {
masters {
id {
type: MASTER
index: 0
}
bound_rpc_address {
host: "127.7.24.126"
port: 39219
}
bound_http_address {
host: "127.7.24.126"
port: 41995
}
}
masters {
id {
type: MASTER
index: 1
}
bound_rpc_address {
host: "127.7.24.125"
port: 35285
}
bound_http_address {
host: "127.7.24.125"
port: 45509
}
}
masters {
id {
type: MASTER
index: 2
}
bound_rpc_address {
host: "127.7.24.124"
port: 41745
}
bound_http_address {
host: "127.7.24.124"
port: 32803
}
}
}
23:40:10.656 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: get_tservers {
}
23:40:10.656 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:10.656 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-admin
23:40:10.657 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-admin"
}
23:40:10.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.656137 7548 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: No bootstrap required, opened a new log
23:40:10.687 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.686652 7548 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:10.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.687553 7548 raft_consensus.cc:374] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
23:40:10.688 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.687839 7548 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ecad18d3e94677b751c668ead6c7ec, State: Initialized, Role: FOLLOWER
23:40:10.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.688881 7548 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:10.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.692150 7559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:10.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:10.693198 7497 tablet.cc:2369] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
23:40:10.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.693313 7559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: This master's current role is: FOLLOWER
23:40:10.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.695608 7548 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: configured and running, proceeding with master startup.
23:40:10.700 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:10 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:10.701 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-admin@KRBTEST.COM:
23:40:10.711 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:10.712 [DEBUG - Time-limited test] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc
23:40:10.713 [DEBUG - Time-limited test] (SecurityUtil.java:119) Logged in as subject: test-admin@KRBTEST.COM
23:40:10.714 [INFO - Time-limited test] (MiniKuduCluster.java:622) Running kinit for user test-user
23:40:10.714 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: kinit {
username: "test-user"
}
23:40:10.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:10 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:10.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Password for test-user@KRBTEST.COM:
23:40:10.774 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:10.781 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:10.781497 7573 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
23:40:10.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:10.782253 7573 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
23:40:10.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.776692 7461 master_runner.cc:186] Error getting master registration for 127.7.24.126:39219: OK, instance_id { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" instance_seqno: 1786405206602529 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
23:40:10.805 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.805205 7461 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
23:40:10.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.886832 7559 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
23:40:10.887 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.887281 7559 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:10.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.889384 7559 leader_election.cc:288] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d2b57669bfc643f3a807faff97cbc52e (127.7.24.126:39219), faca6f033bfd41d4a65bce873a0bfafa (127.7.24.125:35285)
23:40:10.942 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.941511 7453 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 7
23:40:10.958 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:10 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405208, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.125@KRBTEST.COM for kudu/127.7.24.124@KRBTEST.COM
23:40:10.979 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.979110 7453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: Bootstrap starting.
23:40:10.987 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.987299 7453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: Neither blocks nor log segments found. Creating new log.
23:40:10.989 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.989616 7453 log.cc:824] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: Log is configured to *not* fsync() on all Append() calls
23:40:10.995 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:10.995642 7453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: No bootstrap required, opened a new log
23:40:11.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.026767 7453 raft_consensus.cc:348] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:11.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.027639 7453 raft_consensus.cc:374] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
23:40:11.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.028154 7453 raft_consensus.cc:729] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faca6f033bfd41d4a65bce873a0bfafa, State: Initialized, Role: FOLLOWER
23:40:11.030 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.029115 7453 consensus_queue.cc:260] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:11.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.032349 7577 sys_catalog.cc:455] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:11.033 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.033382 7577 sys_catalog.cc:458] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: This master's current role is: FOLLOWER
23:40:11.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.034648 7453 sys_catalog.cc:564] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: configured and running, proceeding with master startup.
23:40:11.035 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.034921 7425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6ecad18d3e94677b751c668ead6c7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faca6f033bfd41d4a65bce873a0bfafa" is_pre_election: true
23:40:11.036 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.036482 7425 raft_consensus.cc:2454] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ecad18d3e94677b751c668ead6c7ec in term 0.
23:40:11.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.038323 7495 leader_election.cc:302] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ecad18d3e94677b751c668ead6c7ec, faca6f033bfd41d4a65bce873a0bfafa; no voters:
23:40:11.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.039628 7559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 0 FOLLOWER]: Leader pre-election won for term 1
23:40:11.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.040050 7559 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
23:40:11.044 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.040438 7559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 0 FOLLOWER]: Advancing to term 1
23:40:11.049 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.048933 7559 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:11.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.055274 7559 leader_election.cc:288] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [CANDIDATE]: Term 1 election: Requested vote from peers d2b57669bfc643f3a807faff97cbc52e (127.7.24.126:39219), faca6f033bfd41d4a65bce873a0bfafa (127.7.24.125:35285)
23:40:11.057 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.057145 7425 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6ecad18d3e94677b751c668ead6c7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "faca6f033bfd41d4a65bce873a0bfafa"
23:40:11.058 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.057801 7425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 0 FOLLOWER]: Advancing to term 1
23:40:11.068 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.067458 7425 raft_consensus.cc:2454] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6ecad18d3e94677b751c668ead6c7ec in term 1.
23:40:11.069 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.069126 7495 leader_election.cc:302] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d6ecad18d3e94677b751c668ead6c7ec, faca6f033bfd41d4a65bce873a0bfafa; no voters:
23:40:11.070 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.070322 7559 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 1 FOLLOWER]: Leader election won for term 1
23:40:11.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.072715 7559 raft_consensus.cc:686] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 1 LEADER]: Becoming Leader. State: Replica: d6ecad18d3e94677b751c668ead6c7ec, State: Running, Role: LEADER
23:40:11.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.075346 7559 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:11.087 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:11.087425 7589 catalog_manager.cc:1558] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
23:40:11.088 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:11.087785 7589 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
23:40:11.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.113672 7588 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ecad18d3e94677b751c668ead6c7ec. Latest consensus state: current_term: 1 leader_uuid: "d6ecad18d3e94677b751c668ead6c7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:11.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.079399 7375 master_runner.cc:186] Error getting master registration for 127.7.24.126:39219: OK, instance_id { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" instance_seqno: 1786405206602529 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
23:40:11.115 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.115676 7375 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
23:40:11.116 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.115860 7588 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: This master's current role is: LEADER
23:40:11.120 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.120646 7592 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
23:40:11.129 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.128806 7592 catalog_manager.cc:1484] Initializing Kudu cluster ID...
23:40:11.160 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.159838 7425 raft_consensus.cc:1261] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 1 FOLLOWER]: Refusing update from remote peer d6ecad18d3e94677b751c668ead6c7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
23:40:11.162 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.161893 7559 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
23:40:11.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:11.221802 7403 tablet.cc:2369] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: Can't schedule compaction. Clean time has not been advanced past its initial value.
23:40:11.228 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.226347 7577 sys_catalog.cc:455] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ecad18d3e94677b751c668ead6c7ec. Latest consensus state: current_term: 1 leader_uuid: "d6ecad18d3e94677b751c668ead6c7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:11.229 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.229009 7577 sys_catalog.cc:458] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: This master's current role is: FOLLOWER
23:40:11.231 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.230939 7559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6ecad18d3e94677b751c668ead6c7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:11.232 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.231714 7559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: This master's current role is: LEADER
23:40:11.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.233860 7592 catalog_manager.cc:1347] Generated new cluster ID: 39e7549b33704811bd4808ceebed0c76
23:40:11.234 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.234223 7592 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
23:40:11.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.256979 7577 sys_catalog.cc:455] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6ecad18d3e94677b751c668ead6c7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:11.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.257714 7577 sys_catalog.cc:458] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: This master's current role is: FOLLOWER
23:40:11.267 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.266525 7592 catalog_manager.cc:1370] Generated new certificate authority record
23:40:11.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.274854 7592 catalog_manager.cc:1504] Loading token signing keys...
23:40:11.301 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.300555 7366 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } attempt: 7
23:40:11.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.320973 7592 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: Generated new TSK 0
23:40:11.325 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.324555 7592 catalog_manager.cc:1514] Initializing in-progress tserver states...
23:40:11.333 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:11 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405206, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.126@KRBTEST.COM for kudu/127.7.24.124@KRBTEST.COM
23:40:11.360 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.360536 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: Bootstrap starting.
23:40:11.368 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.367939 7366 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: Neither blocks nor log segments found. Creating new log.
23:40:11.370 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.370467 7366 log.cc:824] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: Log is configured to *not* fsync() on all Append() calls
23:40:11.381 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.381273 7366 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: No bootstrap required, opened a new log
23:40:11.412 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.412004 7366 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:11.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.413084 7366 raft_consensus.cc:374] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
23:40:11.413 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.413535 7366 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b57669bfc643f3a807faff97cbc52e, State: Initialized, Role: FOLLOWER
23:40:11.415 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.414589 7366 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:11.420 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.419706 7366 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: configured and running, proceeding with master startup.
23:40:11.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.419862 7341 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6ecad18d3e94677b751c668ead6c7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2b57669bfc643f3a807faff97cbc52e" is_pre_election: true
23:40:11.423 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.420786 7341 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6ecad18d3e94677b751c668ead6c7ec in term 0.
23:40:11.425 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.423918 7340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6ecad18d3e94677b751c668ead6c7ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d2b57669bfc643f3a807faff97cbc52e"
23:40:11.426 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.419373 7604 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:11.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.426560 7604 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: This master's current role is: FOLLOWER
23:40:11.427 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.427070 7339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [term 0 FOLLOWER]: Advancing to term 1
23:40:11.437 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.436905 7339 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [term 1 FOLLOWER]: Refusing update from remote peer d6ecad18d3e94677b751c668ead6c7ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
23:40:11.442 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.438935 7559 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
23:40:11.494 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.493444 7604 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6ecad18d3e94677b751c668ead6c7ec. Latest consensus state: current_term: 1 leader_uuid: "d6ecad18d3e94677b751c668ead6c7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:11.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.495498 7604 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: This master's current role is: FOLLOWER
23:40:11.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.494711 7613 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d6ecad18d3e94677b751c668ead6c7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:11.496 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.496012 7613 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: This master's current role is: FOLLOWER
23:40:11.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.521376 7623 catalog_manager.cc:1259] Loaded cluster ID: 39e7549b33704811bd4808ceebed0c76
23:40:11.523 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.522974 7623 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: loading cluster ID for follower catalog manager: success
23:40:11.540 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.539870 7623 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: acquiring CA information for follower catalog manager: success
23:40:11.554 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.550319 7623 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
23:40:11.973 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.972865 7559 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d6ecad18d3e94677b751c668ead6c7ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:11.974 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:11.973796 7559 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: This master's current role is: LEADER
23:40:12.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:12.100903 7589 catalog_manager.cc:1259] Loaded cluster ID: 39e7549b33704811bd4808ceebed0c76
23:40:12.101 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:12.101423 7589 catalog_manager.cc:1552] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: loading cluster ID for follower catalog manager: success
23:40:12.114 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:12.114447 7589 catalog_manager.cc:1574] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: acquiring CA information for follower catalog manager: success
23:40:12.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:12.133049 7589 catalog_manager.cc:1602] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
23:40:13.777 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.7.24.124:41745
23:40:13.778 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 2
}
}
23:40:13.779 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:13.778966 7265 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 7461
23:40:13.818 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:13.818 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.7.24.125:35285
23:40:13.818 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 1
}
}
23:40:13.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:13.820502 7265 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 7375
23:40:13.867 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:13.868 [INFO - Time-limited test] (MiniKuduCluster.java:399) Killing master server 127.7.24.126:39219
23:40:13.868 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: stop_daemon {
id {
type: MASTER
index: 0
}
}
23:40:13.870 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:13.869958 7265 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 7292
23:40:13.908 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:13.908 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.7.24.124:41745
23:40:13.908 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 2
}
}
23:40:13.910 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:13.910260 7265 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/logs
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data/info.pb
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:40:13.912 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.124:41745
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.124
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=32803
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.124.keytab
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.124
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:13.913 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
23:40:14.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.242079 7630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:40:14.242 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.242694 7630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:40:14.243 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.243309 7630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:40:14.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.275888 7630 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
23:40:14.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.276144 7630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
23:40:14.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.276409 7630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:40:14.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.276605 7630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
23:40:14.276 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.276793 7630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
23:40:14.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.276978 7630 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
23:40:14.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.277123 7630 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34253/jwks.json
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:14.315121 7630 master_runner.cc:386] Master server non-default flags:
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.124.keytab
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.124
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:14.315 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.124:41745
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data/info.pb
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.124
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=32803
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/logs
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:40:14.316 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:14.316305 7630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:40:14.317 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:14.317265 7630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:40:14.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.335949 7639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:15.586 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.336596 7637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:15.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:15.587075 7638 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
23:40:15.587 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.587316 7630 server_base.cc:1029] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
23:40:15.588 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:14.335958 7636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:15.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.591907 7630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:40:15.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.595414 7630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:40:15.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.596876 7630 hybrid_clock.cc:648] HybridClock initialized: now 1786405215596828 us; error 52 us; skew 500 ppm
23:40:15.598 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.597816 7630 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:40:15.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:15 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405215, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:15.610 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.609838 7630 init.cc:377] Logged in from keytab as kudu/127.7.24.124@KRBTEST.COM (short username kudu)
23:40:15.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.616374 7630 webserver.cc:460] Webserver started at http://127.7.24.124:32803/ using document root <none> and password file <none>
23:40:15.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.617712 7630 fs_manager.cc:359] Metadata directory not provided
23:40:15.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.618016 7630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:40:15.629 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.629416 7630 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.006s sys 0.000s
23:40:15.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.634609 7649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:40:15.636 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.635871 7630 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
23:40:15.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.636171 7630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal
23:40:15.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d6ecad18d3e94677b751c668ead6c7ec"
23:40:15.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:10 on dist-test-slave-9jjr"
23:40:15.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.638667 7630 fs_report.cc:383] FS layout report
23:40:15.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:40:15.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal
23:40:15.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/wal
23:40:15.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data/data
23:40:15.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:40:15.639 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:40:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:40:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:40:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:40:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:40:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:40:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:40:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:40:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:40:15.640 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:40:15.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.679730 7630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:40:15.685 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.685467 7630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:40:15.686 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.685913 7630 kserver.cc:163] Server-wide thread pool size limit: 3276
23:40:15.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.770254 7630 rpc_server.cc:307] RPC server started. Bound to: 127.7.24.124:41745
23:40:15.770 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.770427 7708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.24.124:41745 every 8 connection(s)
23:40:15.773 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.773100 7630 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-2/data/info.pb
23:40:15.782 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.782414 7265 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 7630
23:40:15.783 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:15.783 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.7.24.125:35285
23:40:15.783 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 1
}
}
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.785097 7265 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/logs
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data/info.pb
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:15.785 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.125:35285
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.125
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45509
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.125.keytab
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.125
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:15.786 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
23:40:15.787 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.785791 7709 sys_catalog.cc:263] Verifying existing consensus state
23:40:15.794 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.794627 7709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: Bootstrap starting.
23:40:15.837 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.836792 7709 log.cc:824] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: Log is configured to *not* fsync() on all Append() calls
23:40:15.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.853798 7709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
23:40:15.854 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.854684 7709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: Bootstrap complete.
23:40:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.885895 7709 raft_consensus.cc:348] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:15.888 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.888465 7709 raft_consensus.cc:729] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6ecad18d3e94677b751c668ead6c7ec, State: Initialized, Role: FOLLOWER
23:40:15.889 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.889231 7709 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:15.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.892098 7714 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:15.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.893113 7714 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: This master's current role is: FOLLOWER
23:40:15.893 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.893523 7709 sys_catalog.cc:564] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: configured and running, proceeding with master startup.
23:40:15.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.925766 7725 catalog_manager.cc:1259] Loaded cluster ID: 39e7549b33704811bd4808ceebed0c76
23:40:15.927 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.925997 7725 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: loading cluster ID for follower catalog manager: success
23:40:15.943 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.942919 7725 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: acquiring CA information for follower catalog manager: success
23:40:15.949 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.949222 7725 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
23:40:15.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.928071 7630 master_runner.cc:186] Error getting master registration for 127.7.24.126:39219: Network error: Client connection negotiation failed: client connection to 127.7.24.126:39219: connect: Connection refused (error 111),
23:40:15.967 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:15.967624 7630 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
23:40:16.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.171103 7711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:40:16.171 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.171705 7711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:40:16.172 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.172328 7711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:40:16.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.207379 7711 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
23:40:16.207 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.207677 7711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
23:40:16.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.208107 7711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:40:16.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.208424 7711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
23:40:16.208 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.208719 7711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
23:40:16.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.209017 7711 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
23:40:16.209 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.209242 7711 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34253/jwks.json
23:40:16.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:16.251191 7711 master_runner.cc:386] Master server non-default flags:
23:40:16.251 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.125.keytab
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.125
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.125:35285
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:16.252 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data/info.pb
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.125
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=45509
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/logs
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:40:16.253 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:16.252709 7711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:40:16.254 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:16.253940 7711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:40:16.275 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.274761 7735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:17.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:16.276664 7711 server_base.cc:1034] running on GCE node
23:40:17.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.277750 7737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:17.590 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:16.276010 7734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:17.595 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.595522 7711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:40:17.599 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.598862 7711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:40:17.600 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.600306 7711 hybrid_clock.cc:648] HybridClock initialized: now 1786405217600240 us; error 82 us; skew 500 ppm
23:40:17.601 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.601091 7711 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:40:17.607 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:17 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405217, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:17.608 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.608057 7711 init.cc:377] Logged in from keytab as kudu/127.7.24.125@KRBTEST.COM (short username kudu)
23:40:17.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.615182 7711 webserver.cc:460] Webserver started at http://127.7.24.125:45509/ using document root <none> and password file <none>
23:40:17.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.616472 7711 fs_manager.cc:359] Metadata directory not provided
23:40:17.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.616883 7711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:40:17.628 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.627786 7711 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s
23:40:17.633 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.633031 7750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:40:17.634 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.634416 7711 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.634716 7711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "faca6f033bfd41d4a65bce873a0bfafa"
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:08 on dist-test-slave-9jjr"
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.637190 7711 fs_report.cc:383] FS layout report
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/wal
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data/data
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:40:17.637 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:40:17.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:40:17.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:40:17.638 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:40:17.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.674301 7711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:40:17.680 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.680312 7711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:40:17.681 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.680804 7711 kserver.cc:163] Server-wide thread pool size limit: 3276
23:40:17.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.690349 7758 raft_consensus.cc:482] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
23:40:17.691 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.690979 7758 raft_consensus.cc:504] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:17.696 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.694491 7758 leader_election.cc:288] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2b57669bfc643f3a807faff97cbc52e (127.7.24.126:39219), faca6f033bfd41d4a65bce873a0bfafa (127.7.24.125:35285)
23:40:17.708 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:17.707811 7650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.24.126:39219: connect: Connection refused (error 111)
23:40:17.726 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:17.725643 7650 leader_election.cc:334] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d2b57669bfc643f3a807faff97cbc52e (127.7.24.126:39219): Network error: Client connection negotiation failed: client connection to 127.7.24.126:39219: connect: Connection refused (error 111)
23:40:17.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:17.728657 7653 leader_election.cc:334] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer faca6f033bfd41d4a65bce873a0bfafa (127.7.24.125:35285): Network error: Client connection negotiation failed: client connection to 127.7.24.125:35285: connect: Connection refused (error 111)
23:40:17.729 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.729096 7653 leader_election.cc:302] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6ecad18d3e94677b751c668ead6c7ec; no voters: d2b57669bfc643f3a807faff97cbc52e, faca6f033bfd41d4a65bce873a0bfafa
23:40:17.730 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.730103 7758 raft_consensus.cc:2726] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
23:40:17.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.810913 7711 rpc_server.cc:307] RPC server started. Bound to: 127.7.24.125:35285
23:40:17.811 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.811060 7811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.24.125:35285 every 8 connection(s)
23:40:17.815 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.814906 7711 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-1/data/info.pb
23:40:17.816 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.815693 7265 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 7711
23:40:17.816 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:17.816 [INFO - Time-limited test] (MiniKuduCluster.java:380) Starting master server 127.7.24.126:39219
23:40:17.816 [DEBUG - Time-limited test] (MiniKuduCluster.java:185) Request: start_daemon {
id {
type: MASTER
index: 0
}
}
23:40:17.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.818470 7265 external_mini_cluster.cc:1326] Running /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:17.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu
23:40:17.820 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --block_manager=log
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=localhost
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/logs
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data/info.pb
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) master
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) run
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.126:39219
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.126
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41995
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.126.keytab
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.126
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=test-admin
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=test-user
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:17.821 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:17.822 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10 with env {KRB5CCNAME=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
23:40:17.840 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.840304 7812 sys_catalog.cc:263] Verifying existing consensus state
23:40:17.848 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.848469 7812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: Bootstrap starting.
23:40:17.905 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.904613 7812 log.cc:824] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: Log is configured to *not* fsync() on all Append() calls
23:40:17.937 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.937285 7812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
23:40:17.938 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.938364 7812 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: Bootstrap complete.
23:40:17.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.976541 7812 raft_consensus.cc:348] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:17.980 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.980023 7812 raft_consensus.cc:729] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: faca6f033bfd41d4a65bce873a0bfafa, State: Initialized, Role: FOLLOWER
23:40:17.981 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.981038 7812 consensus_queue.cc:260] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:17.985 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.984156 7817 sys_catalog.cc:455] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:17.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.986248 7817 sys_catalog.cc:458] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: This master's current role is: FOLLOWER
23:40:17.986 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:17.985865 7812 sys_catalog.cc:564] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: configured and running, proceeding with master startup.
23:40:18.037 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:18.037576 7830 catalog_manager.cc:1259] Loaded cluster ID: 39e7549b33704811bd4808ceebed0c76
23:40:18.038 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:18.038097 7830 catalog_manager.cc:1552] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: loading cluster ID for follower catalog manager: success
23:40:18.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:18.028159 7711 master_runner.cc:186] Error getting master registration for 127.7.24.126:39219: Network error: Client connection negotiation failed: client connection to 127.7.24.126:39219: connect: Connection refused (error 111),
23:40:18.056 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:18.056319 7711 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
23:40:18.061 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:18.060825 7830 catalog_manager.cc:1574] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: acquiring CA information for follower catalog manager: success
23:40:18.067 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:18.067513 7830 catalog_manager.cc:1602] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
23:40:18.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.223801 7814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
23:40:18.224 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.224535 7814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
23:40:18.225 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.225283 7814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
23:40:18.261 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.261673 7814 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=10
23:40:18.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.261951 7814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
23:40:18.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.262470 7814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
23:40:18.262 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.262792 7814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
23:40:18.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.263105 7814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
23:40:18.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.263397 7814 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
23:40:18.263 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.263629 7814 flags.cc:425] Enabled experimental flag: --jwks_url=http://localhost:34253/jwks.json
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:18.303524 7814 master_runner.cc:386] Master server non-default flags:
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_poll_interval_ms=100
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --builtin_ntp_servers=127.7.24.84:35183
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ntp_initial_sync_wait_secs=10
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --time_source=builtin
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_data_dirs=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --fs_wal_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --authn_token_validity_seconds=10
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_ca_key_size=768
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --master_addresses=127.7.24.126:39219,127.7.24.125:35285,127.7.24.124:41745
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --keytab_file=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/kudu_127.7.24.126.keytab
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --principal=kudu/127.7.24.126
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --ipki_server_key_size=768
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --openssl_security_level_override=0
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --tsk_num_rsa_bits=512
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_bind_addresses=127.7.24.126:39219
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_reuseport=true
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_server_allow_ephemeral_ports=true
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_jwt_token_auth=true
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --jwks_url=http://localhost:34253/jwks.json
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --rpc_authentication=required
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --superuser_acl=<redacted>
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --user_acl=<redacted>
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --metrics_log_interval_ms=1000
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_format=pb
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --server_dump_info_path=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data/info.pb
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_interface=127.7.24.126
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --webserver_port=41995
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --never_fsync=true
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --redact=none
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_experimental_flags=true
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --unlock_unsafe_flags=true
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --enable_minidumps=false
23:40:18.304 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --log_dir=/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/logs
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logbuflevel=-1
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --logtostderr=true
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751)
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Master server version:
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) kudu 1.18.0-SNAPSHOT
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build type FASTDEBUG
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) built by systest at 10 Aug 2026 22:47:25 UTC on ccycloud.quasar-hytfkc.root.comops.site
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) build id 4579239
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) TSAN enabled
23:40:18.305 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:18.305068 7814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
23:40:18.306 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:18.306326 7814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
23:40:18.326 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.326179 7839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:19.388 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.388203 7849 raft_consensus.cc:482] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
23:40:19.389 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.388769 7849 raft_consensus.cc:504] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:19.392 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.391912 7849 leader_election.cc:288] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d2b57669bfc643f3a807faff97cbc52e (127.7.24.126:39219), d6ecad18d3e94677b751c668ead6c7ec (127.7.24.124:41745)
23:40:19.407 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:19.406518 7751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.7.24.126:39219: connect: Connection refused (error 111)
23:40:19.414 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:19.413456 7751 leader_election.cc:334] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d2b57669bfc643f3a807faff97cbc52e (127.7.24.126:39219): Network error: Client connection negotiation failed: client connection to 127.7.24.126:39219: connect: Connection refused (error 111)
23:40:19.430 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.429643 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "faca6f033bfd41d4a65bce873a0bfafa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6ecad18d3e94677b751c668ead6c7ec" is_pre_election: true
23:40:19.431 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.430589 7681 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate faca6f033bfd41d4a65bce873a0bfafa in term 1.
23:40:19.432 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.432161 7751 leader_election.cc:302] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d6ecad18d3e94677b751c668ead6c7ec, faca6f033bfd41d4a65bce873a0bfafa; no voters: d2b57669bfc643f3a807faff97cbc52e
23:40:19.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.433022 7849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 1 FOLLOWER]: Leader pre-election won for term 2
23:40:19.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.433380 7849 raft_consensus.cc:482] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
23:40:19.433 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.433691 7849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 1 FOLLOWER]: Advancing to term 2
23:40:19.443 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.442945 7849 raft_consensus.cc:504] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:19.445 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.444917 7849 leader_election.cc:288] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [CANDIDATE]: Term 2 election: Requested vote from peers d2b57669bfc643f3a807faff97cbc52e (127.7.24.126:39219), d6ecad18d3e94677b751c668ead6c7ec (127.7.24.124:41745)
23:40:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.446735 7681 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "faca6f033bfd41d4a65bce873a0bfafa" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6ecad18d3e94677b751c668ead6c7ec"
23:40:19.447 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.447326 7681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 1 FOLLOWER]: Advancing to term 2
23:40:19.449 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:19.449503 7751 leader_election.cc:334] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer d2b57669bfc643f3a807faff97cbc52e (127.7.24.126:39219): Network error: Client connection negotiation failed: client connection to 127.7.24.126:39219: connect: Connection refused (error 111)
23:40:19.457 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.456776 7681 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate faca6f033bfd41d4a65bce873a0bfafa in term 2.
23:40:19.458 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.458276 7751 leader_election.cc:302] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d6ecad18d3e94677b751c668ead6c7ec, faca6f033bfd41d4a65bce873a0bfafa; no voters: d2b57669bfc643f3a807faff97cbc52e
23:40:19.459 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.459221 7849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 2 FOLLOWER]: Leader election won for term 2
23:40:19.462 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.462258 7849 raft_consensus.cc:686] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [term 2 LEADER]: Becoming Leader. State: Replica: faca6f033bfd41d4a65bce873a0bfafa, State: Running, Role: LEADER
23:40:19.464 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.463330 7849 consensus_queue.cc:237] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:19.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.477900 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: SysCatalogTable state changed. Reason: New leader faca6f033bfd41d4a65bce873a0bfafa. Latest consensus state: current_term: 2 leader_uuid: "faca6f033bfd41d4a65bce873a0bfafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:19.478 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.478653 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: This master's current role is: LEADER
23:40:19.481 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.481577 7858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
23:40:19.488 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.488150 7858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
23:40:19.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.491178 7858 catalog_manager.cc:1259] Loaded cluster ID: 39e7549b33704811bd4808ceebed0c76
23:40:19.491 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.491382 7858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
23:40:19.495 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.495234 7858 catalog_manager.cc:1504] Loading token signing keys...
23:40:19.498 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.498406 7858 catalog_manager.cc:5879] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa: Loaded TSK: 0
23:40:19.499 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.499768 7858 catalog_manager.cc:1514] Initializing in-progress tserver states...
23:40:19.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.327375 7838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:19.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:18.328670 7814 server_base.cc:1034] running on GCE node
23:40:19.650 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:18.326375 7842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
23:40:19.655 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.654975 7814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
23:40:19.658 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.658416 7814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
23:40:19.660 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.660163 7814 hybrid_clock.cc:648] HybridClock initialized: now 1786405219660118 us; error 60 us; skew 500 ppm
23:40:19.661 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.661268 7814 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
23:40:19.666 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:19 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405219, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:19.667 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.667688 7814 init.cc:377] Logged in from keytab as kudu/127.7.24.126@KRBTEST.COM (short username kudu)
23:40:19.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.673759 7814 webserver.cc:460] Webserver started at http://127.7.24.126:41995/ using document root <none> and password file <none>
23:40:19.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.674912 7814 fs_manager.cc:359] Metadata directory not provided
23:40:19.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.675179 7814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
23:40:19.684 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.683948 7814 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.007s
23:40:19.689 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.689358 7868 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
23:40:19.690 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.690629 7814 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
23:40:19.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.690997 7814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data,/tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal
23:40:19.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) uuid: "d2b57669bfc643f3a807faff97cbc52e"
23:40:19.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) format_stamp: "Formatted at 2026-08-10 23:40:06 on dist-test-slave-9jjr"
23:40:19.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.693581 7814 fs_report.cc:383] FS layout report
23:40:19.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) --------------------
23:40:19.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) wal directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal
23:40:19.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) metadata directory: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/wal
23:40:19.694 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 1 data directories: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data/data
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live blocks: 0
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes: 0
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total live bytes (after alignment): 0
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total number of LBM containers: 0 (0 full)
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for missing blocks
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Did not check for orphaned blocks
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM containers with extra space: 0 (0 repaired)
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total full LBM container extra space in bytes: 0 (0 repaired)
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total incomplete LBM containers: 0 (0 repaired)
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total LBM partial records: 0 (0 repaired)
23:40:19.695 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
23:40:19.757 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.757385 7814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
23:40:19.765 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.765169 7814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
23:40:19.766 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.765753 7814 kserver.cc:163] Server-wide thread pool size limit: 3276
23:40:19.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.890026 7814 rpc_server.cc:307] RPC server started. Bound to: 127.7.24.126:39219
23:40:19.890 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.890230 7921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.7.24.126:39219 every 8 connection(s)
23:40:19.894 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.894023 7814 server_base.cc:1166] Dumped server information to /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/master-0/data/info.pb
23:40:19.899 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.898993 7265 external_mini_cluster.cc:1388] Started /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu as pid 7814
23:40:19.899 [DEBUG - Time-limited test] (MiniKuduCluster.java:195) Response:
23:40:19.905 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.7.24.124:41745(127.7.24.124:41745)] connecting to peer
23:40:19.906 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:19.907 [DEBUG - Time-limited test] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:19.916 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.124:41745(127.7.24.124:41745)] Successfully connected to peer
23:40:19.919 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:19.920 [DEBUG - kudu-nio-0] (SecurityContext.java:225) Refreshing Kerberos credentials...
23:40:19.920 [DEBUG - kudu-nio-0] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc
23:40:19.922 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.913672 7922 sys_catalog.cc:263] Verifying existing consensus state
23:40:19.926 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:19 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405210, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:19.926 [DEBUG - kudu-nio-0] (SecurityUtil.java:119) Logged in as subject: test-user@KRBTEST.COM
23:40:19.926 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:19.927 [ERROR - kudu-nio-0] (SecurityContext.java:249) Attempted to refresh Kerberos credentials from ticket cache but found that the new Kerberos principal test-user@KRBTEST.COM did not match the original principal test-admin@KRBTEST.COM. Ignoring.
23:40:19.928 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.7.24.124
23:40:19.948 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:19 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.7.24.124@KRBTEST.COM
23:40:19.966 [DEBUG - kudu-nio-0] (Negotiator.java:919) Authenticated connection [id: 0x39f90caa, L:/127.0.0.1:35110 - R:/127.7.24.124:41745] using SASL/GSSAPI
23:40:19.990 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:19.990401 7922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: Bootstrap starting.
23:40:19.997 [DEBUG - kudu-nio-1] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.7.24.125
23:40:19.998 [DEBUG - kudu-nio-0] (Negotiator.java:529) SASL mechanism GSSAPI chosen for peer 127.7.24.126
23:40:20.015 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:20 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405217, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.125@KRBTEST.COM for kudu/127.7.24.126@KRBTEST.COM
23:40:20.022 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
23:40:20.022 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
23:40:20.023 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:20.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.026530 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.925821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59794 (local address 127.7.24.125:35285)
23:40:20.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.993443 (+ 67622us) server_negotiation.cc:206] Beginning negotiation
23:40:20.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.993476 (+ 33us) server_negotiation.cc:399] Waiting for connection header
23:40:20.026 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.993540 (+ 64us) server_negotiation.cc:407] Connection header received
23:40:20.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.993842 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.027 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.993873 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.994480 (+ 607us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.996552 (+ 2072us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.004669 (+ 8117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
23:40:20.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.006931 (+ 2262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
23:40:20.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.018592 (+ 11661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
23:40:20.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.019931 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
23:40:20.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.020205 (+ 274us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
23:40:20.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.024844 (+ 4639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59794: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:59794): Cannot send after transport endpoint shutdown (error 108)
23:40:20.028 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":67115,"thread_start_us":66768,"threads_started":1}
23:40:20.030 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
23:40:20.039 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: Server requires Kerberos, but this client is not authenticated (missing or expired TGT)
23:40:20.040 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:20.043 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.044871 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.916183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46330 (local address 127.7.24.126:39219)
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.979346 (+ 63163us) server_negotiation.cc:206] Beginning negotiation
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.979385 (+ 39us) server_negotiation.cc:399] Waiting for connection header
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.979462 (+ 77us) server_negotiation.cc:407] Connection header received
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.996843 (+ 17381us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.996903 (+ 60us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.997437 (+ 534us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:19.997905 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.002480 (+ 4575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.005621 (+ 3141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.026669 (+ 21048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.027642 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.028118 (+ 476us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.041998 (+ 13880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46330: BlockingRecv error: failed to read from TLS socket (remote: 127.0.0.1:46330): Cannot send after transport endpoint shutdown (error 108)
23:40:20.045 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"mutex_wait_us":68,"server-negotiator.queue_time_us":62629,"thread_start_us":69595,"threads_started":1}
23:40:20.052 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:20.053 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:20.054 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:20.058 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.059 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.061 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:20.064 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.064985 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.053962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59806 (local address 127.7.24.125:35285)
23:40:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.055828 (+ 1866us) server_negotiation.cc:206] Beginning negotiation
23:40:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.055892 (+ 64us) server_negotiation.cc:399] Waiting for connection header
23:40:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.055981 (+ 89us) server_negotiation.cc:407] Connection header received
23:40:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.056318 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.056374 (+ 56us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.065 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.056873 (+ 499us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.059247 (+ 2374us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.062244 (+ 2997us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59806: BlockingRecv error: recv got EOF from 127.0.0.1:59806 (error 108)
23:40:20.066 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1195}
23:40:20.069 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.069 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.069 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.072202 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.062968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.7.24.126:39219)
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.063499 (+ 531us) server_negotiation.cc:206] Beginning negotiation
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.063528 (+ 29us) server_negotiation.cc:399] Waiting for connection header
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.064291 (+ 763us) server_negotiation.cc:407] Connection header received
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.066828 (+ 2537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.066857 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.067097 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.067254 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.071218 (+ 3964us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46346: BlockingRecv error: recv got EOF from 127.0.0.1:46346 (error 108)
23:40:20.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":126}
23:40:20.073 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:20.077 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.077353 7922 log.cc:824] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: Log is configured to *not* fsync() on all Append() calls
23:40:20.085 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:20.086 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:20.086 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:20.090 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.090 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.091 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.092618 7681 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [term 2 FOLLOWER]: Refusing update from remote peer faca6f033bfd41d4a65bce873a0bfafa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.092803 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.087290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.7.24.125:35285)
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.087904 (+ 614us) server_negotiation.cc:206] Beginning negotiation
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.087933 (+ 29us) server_negotiation.cc:399] Waiting for connection header
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.089042 (+ 1109us) server_negotiation.cc:407] Connection header received
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.089205 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.089231 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.089480 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.089702 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.091758 (+ 2056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59822: BlockingRecv error: recv got EOF from 127.0.0.1:59822 (error 108)
23:40:20.094 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":139}
23:40:20.096 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.094559 7855 consensus_queue.cc:1035] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
23:40:20.096 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:20.099 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.100 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.100 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.102392 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.091357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.7.24.126:39219)
23:40:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.093828 (+ 2471us) server_negotiation.cc:206] Beginning negotiation
23:40:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.093853 (+ 25us) server_negotiation.cc:399] Waiting for connection header
23:40:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.097229 (+ 3376us) server_negotiation.cc:407] Connection header received
23:40:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.098475 (+ 1246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.098505 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.098766 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.098908 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.102 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.101391 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46360: BlockingRecv error: recv got EOF from 127.0.0.1:46360 (error 108)
23:40:20.103 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2058}
23:40:20.105 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:20.106 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:20.107 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:20.108 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:20.108 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:20.110 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.110 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.110 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.112366 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.107982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59824 (local address 127.7.24.125:35285)
23:40:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.108502 (+ 520us) server_negotiation.cc:206] Beginning negotiation
23:40:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.108526 (+ 24us) server_negotiation.cc:399] Waiting for connection header
23:40:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.108572 (+ 46us) server_negotiation.cc:407] Connection header received
23:40:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.108704 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.108729 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.108943 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.109153 (+ 210us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.112 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.111390 (+ 2237us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59824: BlockingRecv error: recv got EOF from 127.0.0.1:59824 (error 108)
23:40:20.113 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":131}
23:40:20.114 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.115 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.117 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.117749 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.109834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46372 (local address 127.7.24.126:39219)
23:40:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.110397 (+ 563us) server_negotiation.cc:206] Beginning negotiation
23:40:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.110419 (+ 22us) server_negotiation.cc:399] Waiting for connection header
23:40:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.110465 (+ 46us) server_negotiation.cc:407] Connection header received
23:40:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.110634 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.110659 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.110879 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.111057 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.116760 (+ 5703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46372: BlockingRecv error: recv got EOF from 127.0.0.1:46372 (error 108)
23:40:20.118 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":146}
23:40:20.119 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:20.132 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.131757 7922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
23:40:20.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.132769 7922 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: Bootstrap complete.
23:40:20.140 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.139882 7932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader faca6f033bfd41d4a65bce873a0bfafa. Latest consensus state: current_term: 2 leader_uuid: "faca6f033bfd41d4a65bce873a0bfafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:20.141 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.140730 7932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: This master's current role is: FOLLOWER
23:40:20.143 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:20.165 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:20.166 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:20.166 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:20.167 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:20.171 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.171 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.171 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.171 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.172 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:20.172 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.173717 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.168402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46384 (local address 127.7.24.126:39219)
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.170396 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.170423 (+ 27us) server_negotiation.cc:399] Waiting for connection header
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.170465 (+ 42us) server_negotiation.cc:407] Connection header received
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.170589 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.170614 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.170822 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.171000 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.172772 (+ 1772us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46384: BlockingRecv error: recv got EOF from 127.0.0.1:46384 (error 108)
23:40:20.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1506}
23:40:20.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.170977 7922 raft_consensus.cc:348] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:20.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.175387 7922 raft_consensus.cc:729] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2b57669bfc643f3a807faff97cbc52e, State: Initialized, Role: FOLLOWER
23:40:20.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.176155 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.167971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.7.24.125:35285)
23:40:20.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.168951 (+ 980us) server_negotiation.cc:206] Beginning negotiation
23:40:20.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.168978 (+ 27us) server_negotiation.cc:399] Waiting for connection header
23:40:20.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.169024 (+ 46us) server_negotiation.cc:407] Connection header received
23:40:20.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.169174 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.176 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.169202 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.169425 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.169602 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.175192 (+ 5590us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59840: BlockingRecv error: recv got EOF from 127.0.0.1:59840 (error 108)
23:40:20.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":561}
23:40:20.177 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.176411 7922 consensus_queue.cc:260] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } }
23:40:20.181 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.180372 7932 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "faca6f033bfd41d4a65bce873a0bfafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:20.182 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.181073 7932 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6ecad18d3e94677b751c668ead6c7ec [sys.catalog]: This master's current role is: FOLLOWER
23:40:20.182 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:20.192 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.191963 7897 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [term 1 FOLLOWER]: Advancing to term 2
23:40:20.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.198354 7855 sys_catalog.cc:455] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "faca6f033bfd41d4a65bce873a0bfafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:20.199 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.199177 7855 sys_catalog.cc:458] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: This master's current role is: LEADER
23:40:20.202 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.202286 7922 sys_catalog.cc:564] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: configured and running, proceeding with master startup.
23:40:20.204 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.203895 7897 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [term 2 FOLLOWER]: Refusing update from remote peer faca6f033bfd41d4a65bce873a0bfafa: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
23:40:20.205 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:20.205 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:20.206 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.205988 7855 consensus_queue.cc:1035] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
23:40:20.207 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:20.208 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.214623 7936 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:20.215 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.215698 7936 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: This master's current role is: FOLLOWER
23:40:20.219 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.219 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.220 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:20.221 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.221531 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.221 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.216438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59854 (local address 127.7.24.125:35285)
23:40:20.222 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.217286 (+ 848us) server_negotiation.cc:206] Beginning negotiation
23:40:20.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.217327 (+ 41us) server_negotiation.cc:399] Waiting for connection header
23:40:20.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.217394 (+ 67us) server_negotiation.cc:407] Connection header received
23:40:20.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.217602 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.217650 (+ 48us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.217986 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.218219 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.220651 (+ 2432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59854: BlockingRecv error: recv got EOF from 127.0.0.1:59854 (error 108)
23:40:20.222 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":187}
23:40:20.224 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:20.225 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:20.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.226197 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.218391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46392 (local address 127.7.24.126:39219)
23:40:20.226 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.219236 (+ 845us) server_negotiation.cc:206] Beginning negotiation
23:40:20.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.219272 (+ 36us) server_negotiation.cc:399] Waiting for connection header
23:40:20.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.219329 (+ 57us) server_negotiation.cc:407] Connection header received
23:40:20.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.219493 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.219530 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.219855 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.220070 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.224930 (+ 4860us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46392: BlockingRecv error: recv got EOF from 127.0.0.1:46392 (error 108)
23:40:20.227 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":304}
23:40:20.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.237280 7936 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "faca6f033bfd41d4a65bce873a0bfafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:20.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.238322 7936 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: This master's current role is: FOLLOWER
23:40:20.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.237977 7943 sys_catalog.cc:455] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: SysCatalogTable state changed. Reason: New leader faca6f033bfd41d4a65bce873a0bfafa. Latest consensus state: current_term: 2 leader_uuid: "faca6f033bfd41d4a65bce873a0bfafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:20.239 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.238776 7943 sys_catalog.cc:458] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e [sys.catalog]: This master's current role is: FOLLOWER
23:40:20.246 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:20.246 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:20.247 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:20.250 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:20.253 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.253 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.253 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.254 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.254 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:20.254 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:20.255 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.255690 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.248621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59868 (local address 127.7.24.125:35285)
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.249104 (+ 483us) server_negotiation.cc:206] Beginning negotiation
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.249127 (+ 23us) server_negotiation.cc:399] Waiting for connection header
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.249172 (+ 45us) server_negotiation.cc:407] Connection header received
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.249306 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.249330 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.249537 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.249713 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.254784 (+ 5071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59868: BlockingRecv error: recv got EOF from 127.0.0.1:59868 (error 108)
23:40:20.256 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":115}
23:40:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.257331 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.251795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46394 (local address 127.7.24.126:39219)
23:40:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.252272 (+ 477us) server_negotiation.cc:206] Beginning negotiation
23:40:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.252295 (+ 23us) server_negotiation.cc:399] Waiting for connection header
23:40:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.252339 (+ 44us) server_negotiation.cc:407] Connection header received
23:40:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.252461 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.252486 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.252700 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.252849 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.257 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.256281 (+ 3432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46394: BlockingRecv error: recv got EOF from 127.0.0.1:46394 (error 108)
23:40:20.258 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":113}
23:40:20.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.277172 7957 catalog_manager.cc:1259] Loaded cluster ID: 39e7549b33704811bd4808ceebed0c76
23:40:20.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.277434 7957 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: loading cluster ID for follower catalog manager: success
23:40:20.284 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:20 dist-test-slave-9jjr krb5kdc[7277](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405219, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.126@KRBTEST.COM for kudu/127.7.24.126@KRBTEST.COM
23:40:20.285 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.285243 7957 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: acquiring CA information for follower catalog manager: success
23:40:20.291 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.291172 7957 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d2b57669bfc643f3a807faff97cbc52e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
23:40:20.325 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:20.325 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:20.326 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:20.326 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:20.332 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.332 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.332 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:20.333 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.333 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.334 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:20.334 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:20.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.334448 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.330482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46420 (local address 127.7.24.126:39219)
23:40:20.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.331029 (+ 547us) server_negotiation.cc:206] Beginning negotiation
23:40:20.335 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.331053 (+ 24us) server_negotiation.cc:399] Waiting for connection header
23:40:20.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.331103 (+ 50us) server_negotiation.cc:407] Connection header received
23:40:20.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.331233 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.331257 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.331481 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.331673 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.333431 (+ 1758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46420: BlockingRecv error: recv got EOF from 127.0.0.1:46420 (error 108)
23:40:20.336 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":129}
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.337301 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.327936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59892 (local address 127.7.24.125:35285)
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.332437 (+ 4501us) server_negotiation.cc:206] Beginning negotiation
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.332462 (+ 25us) server_negotiation.cc:399] Waiting for connection header
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.332509 (+ 47us) server_negotiation.cc:407] Connection header received
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.332645 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.332677 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.332922 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.335798 (+ 2876us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.336467 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59892: BlockingRecv error: recv got EOF from 127.0.0.1:59892 (error 108)
23:40:20.337 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":4093}
23:40:20.585 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:20.586 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:20.589 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.590 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.590 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:20.591 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.591902 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.586924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59904 (local address 127.7.24.125:35285)
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.588583 (+ 1659us) server_negotiation.cc:206] Beginning negotiation
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.588607 (+ 24us) server_negotiation.cc:399] Waiting for connection header
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.588654 (+ 47us) server_negotiation.cc:407] Connection header received
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.588786 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.588811 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.589024 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.590249 (+ 1225us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.592 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.590927 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59904: BlockingRecv error: recv got EOF from 127.0.0.1:59904 (error 108)
23:40:20.592 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1250}
23:40:20.594 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.594 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.594 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:20.595 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:20.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.596405 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.592069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46434 (local address 127.7.24.126:39219)
23:40:20.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.592642 (+ 573us) server_negotiation.cc:206] Beginning negotiation
23:40:20.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.592675 (+ 33us) server_negotiation.cc:399] Waiting for connection header
23:40:20.596 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.592722 (+ 47us) server_negotiation.cc:407] Connection header received
23:40:20.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.593279 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.593308 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.593528 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.593713 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.595673 (+ 1960us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46434: BlockingRecv error: recv got EOF from 127.0.0.1:46434 (error 108)
23:40:20.597 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":155}
23:40:20.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.662266 7934 sys_catalog.cc:455] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "faca6f033bfd41d4a65bce873a0bfafa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d2b57669bfc643f3a807faff97cbc52e" member_type: VOTER last_known_addr { host: "127.7.24.126" port: 39219 } } peers { permanent_uuid: "faca6f033bfd41d4a65bce873a0bfafa" member_type: VOTER last_known_addr { host: "127.7.24.125" port: 35285 } } peers { permanent_uuid: "d6ecad18d3e94677b751c668ead6c7ec" member_type: VOTER last_known_addr { host: "127.7.24.124" port: 41745 } } }
23:40:20.663 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:20.663115 7934 sys_catalog.cc:458] T 00000000000000000000000000000000 P faca6f033bfd41d4a65bce873a0bfafa [sys.catalog]: This master's current role is: LEADER
23:40:20.685 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:20.686 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:20.687 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:20.687 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:20.689 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.690 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.690 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:20.690 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.691 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:20.691 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:20.691 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.692073 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.687427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.7.24.126:39219)
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.688392 (+ 965us) server_negotiation.cc:206] Beginning negotiation
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.688426 (+ 34us) server_negotiation.cc:399] Waiting for connection header
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.688477 (+ 51us) server_negotiation.cc:407] Connection header received
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.688620 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.688648 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.688877 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.689051 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.691108 (+ 2057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46444: BlockingRecv error: recv got EOF from 127.0.0.1:46444 (error 108)
23:40:20.692 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":523}
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:20.693325 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.688208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.7.24.125:35285)
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.689724 (+ 1516us) server_negotiation.cc:206] Beginning negotiation
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.689743 (+ 19us) server_negotiation.cc:399] Waiting for connection header
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.689780 (+ 37us) server_negotiation.cc:407] Connection header received
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.689953 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.689982 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.690231 (+ 249us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.690408 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:20.692411 (+ 2003us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59916: BlockingRecv error: recv got EOF from 127.0.0.1:59916 (error 108)
23:40:20.693 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1195}
23:40:21.065 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:21.066 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:21.066 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:21.066 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:21.069 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:21.070 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:21.070 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:21.070 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:21.070 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:21.071 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:21.071 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:21.071889 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.067535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59932 (local address 127.7.24.125:35285)
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.068439 (+ 904us) server_negotiation.cc:206] Beginning negotiation
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.068466 (+ 27us) server_negotiation.cc:399] Waiting for connection header
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.068516 (+ 50us) server_negotiation.cc:407] Connection header received
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.068666 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.068694 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.068916 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.070311 (+ 1395us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.070996 (+ 685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59932: BlockingRecv error: recv got EOF from 127.0.0.1:59932 (error 108)
23:40:21.072 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":475}
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:21.073233 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.068419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46462 (local address 127.7.24.126:39219)
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.068985 (+ 566us) server_negotiation.cc:206] Beginning negotiation
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.069010 (+ 25us) server_negotiation.cc:399] Waiting for connection header
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.069054 (+ 44us) server_negotiation.cc:407] Connection header received
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.069192 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.069216 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.069845 (+ 629us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.071641 (+ 1796us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.072277 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46462: BlockingRecv error: recv got EOF from 127.0.0.1:46462 (error 108)
23:40:21.073 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":153}
23:40:21.525 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:21.526 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:21.527 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:21.529 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:21.530 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:21.531 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:21.531 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:21.531 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:21.531 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:21.531 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:21.532 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:21.534 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:21.534538 7930 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:21.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.528864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46466 (local address 127.7.24.126:39219)
23:40:21.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.529391 (+ 527us) server_negotiation.cc:206] Beginning negotiation
23:40:21.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.529421 (+ 30us) server_negotiation.cc:399] Waiting for connection header
23:40:21.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.529467 (+ 46us) server_negotiation.cc:407] Connection header received
23:40:21.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.529608 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:21.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.529636 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:21.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.529937 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:21.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.530988 (+ 1051us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:21.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.533132 (+ 2144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46466: BlockingRecv error: recv got EOF from 127.0.0.1:46466 (error 108)
23:40:21.535 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":135}
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:21.535261 7929 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.529872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59944 (local address 127.7.24.125:35285)
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.530438 (+ 566us) server_negotiation.cc:206] Beginning negotiation
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.530463 (+ 25us) server_negotiation.cc:399] Waiting for connection header
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.530506 (+ 43us) server_negotiation.cc:407] Connection header received
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.530639 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.530663 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.530938 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.532333 (+ 1395us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:21.533011 (+ 678us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59944: BlockingRecv error: recv got EOF from 127.0.0.1:59944 (error 108)
23:40:21.536 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":133}
23:40:25.125 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:25.126 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:25.126 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:25.126 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:25.130 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:25.131 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:25.131 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:25.131 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:25.131 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:25.131 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:25.132 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:25.132956 7971 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.126738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59954 (local address 127.7.24.125:35285)
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.129515 (+ 2777us) server_negotiation.cc:206] Beginning negotiation
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.129550 (+ 35us) server_negotiation.cc:399] Waiting for connection header
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.129605 (+ 55us) server_negotiation.cc:407] Connection header received
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.129753 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.129778 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.130008 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.130243 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.131844 (+ 1601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59954: BlockingRecv error: recv got EOF from 127.0.0.1:59954 (error 108)
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2255,"thread_start_us":2693,"threads_started":1}
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:25.133575 7972 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:25.133 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.127484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46478 (local address 127.7.24.126:39219)
23:40:25.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.129988 (+ 2504us) server_negotiation.cc:206] Beginning negotiation
23:40:25.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.130010 (+ 22us) server_negotiation.cc:399] Waiting for connection header
23:40:25.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.130045 (+ 35us) server_negotiation.cc:407] Connection header received
23:40:25.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.130209 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:25.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.130232 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:25.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.130463 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:25.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.130813 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:25.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:25.132439 (+ 1626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46478: BlockingRecv error: recv got EOF from 127.0.0.1:46478 (error 108)
23:40:25.134 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2015,"thread_start_us":2131,"threads_started":1}
23:40:25.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:25 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405225, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:25.614 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:25.614223 7643 init.cc:295] Successfully reacquired a new kerberos TGT
23:40:26.320 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:26.320276 7704 debug-util.cc:398] Leaking SignalData structure 0x7b080008cfa0 after lost signal to thread 7631
23:40:26.321 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:26.320883 7704 debug-util.cc:398] Leaking SignalData structure 0x7b0800073e40 after lost signal to thread 7708
23:40:26.725 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:26.727 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:26.733 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:26.735 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:26.746 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:26.746 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:26.747 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:26.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:26.748477 7978 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:26.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.734745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42634 (local address 127.7.24.126:39219)
23:40:26.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.744866 (+ 10121us) server_negotiation.cc:206] Beginning negotiation
23:40:26.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.744892 (+ 26us) server_negotiation.cc:399] Waiting for connection header
23:40:26.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.744950 (+ 58us) server_negotiation.cc:407] Connection header received
23:40:26.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.745105 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:26.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.745134 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:26.748 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.745372 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:26.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.745593 (+ 221us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:26.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.747280 (+ 1687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42634: BlockingRecv error: recv got EOF from 127.0.0.1:42634 (error 108)
23:40:26.749 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:26.749 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":9160,"thread_start_us":8860,"threads_started":1}
23:40:26.749 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:26.749 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:26.750 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:26.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:26.753396 7977 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:26.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.729822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.7.24.125:35285)
23:40:26.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.745169 (+ 15347us) server_negotiation.cc:206] Beginning negotiation
23:40:26.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.745200 (+ 31us) server_negotiation.cc:399] Waiting for connection header
23:40:26.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.745267 (+ 67us) server_negotiation.cc:407] Connection header received
23:40:26.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.745509 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:26.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.747960 (+ 2451us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:26.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.748298 (+ 338us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:26.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.750844 (+ 2546us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:26.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:26.751866 (+ 1022us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48268: BlockingRecv error: recv got EOF from 127.0.0.1:48268 (error 108)
23:40:26.754 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":14723,"thread_start_us":14332,"threads_started":1}
23:40:27.612 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:27 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405227, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:27.613 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:27.613376 7745 init.cc:295] Successfully reacquired a new kerberos TGT
23:40:28.045 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:28.046 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:28.046 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:28.046 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:28.051 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:28.051 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:28.052 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:28.052 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:28.052 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:28.052 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:28.053 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:28.053836 7979 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.047847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48278 (local address 127.7.24.125:35285)
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.050361 (+ 2514us) server_negotiation.cc:206] Beginning negotiation
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.050441 (+ 80us) server_negotiation.cc:399] Waiting for connection header
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.050498 (+ 57us) server_negotiation.cc:407] Connection header received
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.050647 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.050680 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.050925 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.051136 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.052595 (+ 1459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48278: BlockingRecv error: recv got EOF from 127.0.0.1:48278 (error 108)
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1993,"thread_start_us":1645,"threads_started":1}
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:28.054507 7980 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.048835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.7.24.126:39219)
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.050870 (+ 2035us) server_negotiation.cc:206] Beginning negotiation
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.050885 (+ 15us) server_negotiation.cc:399] Waiting for connection header
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.050920 (+ 35us) server_negotiation.cc:407] Connection header received
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.051028 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.051044 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.051307 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.051687 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:28.054 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:28.053251 (+ 1564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42638: BlockingRecv error: recv got EOF from 127.0.0.1:42638 (error 108)
23:40:28.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1719,"thread_start_us":1400,"threads_started":1}
23:40:29.671 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:29 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405229, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:29.672 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:29.672307 7861 init.cc:295] Successfully reacquired a new kerberos TGT
23:40:30.045 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:30.046 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:30.046 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:30.047 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:30.049 [DEBUG - kudu-nio-1] (SecurityContext.java:225) Refreshing Kerberos credentials...
23:40:30.050 [DEBUG - kudu-nio-1] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc
23:40:30.052 [DEBUG - kudu-nio-1] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
23:40:30.052 [WARN - kudu-nio-1] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
23:40:30.052 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:30.052 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:30.052 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:30.052 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:30.053 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:30.053 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:30.053 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:30.054894 7988 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.048262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42642 (local address 127.7.24.126:39219)
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.050045 (+ 1783us) server_negotiation.cc:206] Beginning negotiation
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.050066 (+ 21us) server_negotiation.cc:399] Waiting for connection header
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.050113 (+ 47us) server_negotiation.cc:407] Connection header received
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.050349 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.050366 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.050551 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.050757 (+ 206us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.053531 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42642: BlockingRecv error: recv got EOF from 127.0.0.1:42642 (error 108)
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1250,"thread_start_us":905,"threads_started":1}
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:30.055079 7987 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.047035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48290 (local address 127.7.24.125:35285)
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.048898 (+ 1863us) server_negotiation.cc:206] Beginning negotiation
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.048924 (+ 26us) server_negotiation.cc:399] Waiting for connection header
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.048977 (+ 53us) server_negotiation.cc:407] Connection header received
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.049127 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.049157 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.049390 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.049598 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:30.053824 (+ 4226us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48290: BlockingRecv error: recv got EOF from 127.0.0.1:48290 (error 108)
23:40:30.055 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1431,"thread_start_us":1485,"threads_started":1}
23:40:30.752 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:30.752454 6714 debug-util.cc:398] Leaking SignalData structure 0x7b08000a13a0 after lost signal to thread 6634
23:40:30.753 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:30.753046 6714 debug-util.cc:398] Leaking SignalData structure 0x7b08000850a0 after lost signal to thread 6718
23:40:33.525 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:33.526 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:33.526 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:33.527 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:33.531 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:33.531 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:33.531 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:33.531 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:33.532 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:33.532 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:33.533 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:33.533607 7999 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.528163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48304 (local address 127.7.24.125:35285)
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.529905 (+ 1742us) server_negotiation.cc:206] Beginning negotiation
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.529942 (+ 37us) server_negotiation.cc:399] Waiting for connection header
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.529985 (+ 43us) server_negotiation.cc:407] Connection header received
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.530219 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.530247 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.530537 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.530749 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.532338 (+ 1589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48304: BlockingRecv error: recv got EOF from 127.0.0.1:48304 (error 108)
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1127,"thread_start_us":733,"threads_started":1}
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:33.534984 8000 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.528311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42648 (local address 127.7.24.126:39219)
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.530453 (+ 2142us) server_negotiation.cc:206] Beginning negotiation
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.530474 (+ 21us) server_negotiation.cc:399] Waiting for connection header
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.530522 (+ 48us) server_negotiation.cc:407] Connection header received
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.530662 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.530691 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.530925 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.531304 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:33.533761 (+ 2457us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42648: BlockingRecv error: recv got EOF from 127.0.0.1:42648 (error 108)
23:40:33.538 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1667,"thread_start_us":1327,"threads_started":1}
23:40:33.732 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:33.731943 7245 debug-util.cc:398] Leaking SignalData structure 0x7b08000af0c0 after lost signal to thread 7107
23:40:33.733 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:33.733012 7245 debug-util.cc:398] Leaking SignalData structure 0x7b08000af220 after lost signal to thread 7249
23:40:35.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:35 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405235, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:35.617 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:35.616707 7643 init.cc:295] Successfully reacquired a new kerberos TGT
23:40:35.945 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:35.946 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:35.946 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:35.947 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:35.952 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:35.952 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:35.952 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:35.952 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:35.952 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:35.953 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:35.953 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:35.953958 8003 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.947739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48320 (local address 127.7.24.125:35285)
23:40:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.950815 (+ 3076us) server_negotiation.cc:206] Beginning negotiation
23:40:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.950842 (+ 27us) server_negotiation.cc:399] Waiting for connection header
23:40:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.950888 (+ 46us) server_negotiation.cc:407] Connection header received
23:40:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.951036 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.951073 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.951326 (+ 253us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:35.954 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.951540 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.952925 (+ 1385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48320: BlockingRecv error: recv got EOF from 127.0.0.1:48320 (error 108)
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2646,"thread_start_us":2806,"threads_started":1}
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:35.954814 8004 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.948194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42662 (local address 127.7.24.126:39219)
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.951752 (+ 3558us) server_negotiation.cc:206] Beginning negotiation
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.951776 (+ 24us) server_negotiation.cc:399] Waiting for connection header
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.951825 (+ 49us) server_negotiation.cc:407] Connection header received
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.951963 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.951981 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.952141 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.952277 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:35.953756 (+ 1479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42662: BlockingRecv error: recv got EOF from 127.0.0.1:42662 (error 108)
23:40:35.955 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":3108,"thread_start_us":2830,"threads_started":1}
23:40:37.615 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:37 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405237, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:37.616 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:37.616278 7745 init.cc:295] Successfully reacquired a new kerberos TGT
23:40:38.345 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:38.346 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:38.346 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:38.347 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:38.351 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:38.351 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:38.351 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:38.351 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:38.352 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:38.352 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:38.352 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:38.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:38.353523 8009 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:38.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.347188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46838 (local address 127.7.24.125:35285)
23:40:38.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.350167 (+ 2979us) server_negotiation.cc:206] Beginning negotiation
23:40:38.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.350199 (+ 32us) server_negotiation.cc:399] Waiting for connection header
23:40:38.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.350269 (+ 70us) server_negotiation.cc:407] Connection header received
23:40:38.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.350438 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:38.353 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.350454 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:38.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.350634 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:38.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.350842 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:38.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.352455 (+ 1613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46838: BlockingRecv error: recv got EOF from 127.0.0.1:46838 (error 108)
23:40:38.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2337,"thread_start_us":1933,"threads_started":1}
23:40:38.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:38.354481 8010 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:38.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.349149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56522 (local address 127.7.24.126:39219)
23:40:38.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.350714 (+ 1565us) server_negotiation.cc:206] Beginning negotiation
23:40:38.354 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.350744 (+ 30us) server_negotiation.cc:399] Waiting for connection header
23:40:38.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.350811 (+ 67us) server_negotiation.cc:407] Connection header received
23:40:38.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.351087 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:38.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.351113 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:38.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.351407 (+ 294us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:38.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.351647 (+ 240us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:38.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:38.353174 (+ 1527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56522: BlockingRecv error: recv got EOF from 127.0.0.1:56522 (error 108)
23:40:38.355 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1002,"thread_start_us":583,"threads_started":1}
23:40:39.674 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:39 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405239, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:39.675 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:39.675036 7861 init.cc:295] Successfully reacquired a new kerberos TGT
23:40:40.985 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:40.986 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:40.986 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:40.987 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:40.989 [DEBUG - kudu-nio-1] (SecurityContext.java:225) Refreshing Kerberos credentials...
23:40:40.990 [DEBUG - kudu-nio-1] (SecurityUtil.java:105) Using ticketCache: /tmp/dist-test-tasksngkcS/test-tmp/mini-kudu-cluster2835394333510724588/krb5kdc/krb5cc
23:40:40.991 [DEBUG - kudu-nio-1] (SecurityUtil.java:122) Could not login via JAAS. Using no credentials: Unable to obtain Principal Name for authentication
23:40:40.991 [WARN - kudu-nio-1] (SecurityContext.java:234) Tried to refresh Kerberos credentials but was unable to re-login from ticket cache
23:40:40.991 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:40.991 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:40.991 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:40.991 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:40.992 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:40.992 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:40.992 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:40.993188 8014 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.987343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46850 (local address 127.7.24.125:35285)
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.988780 (+ 1437us) server_negotiation.cc:206] Beginning negotiation
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.988803 (+ 23us) server_negotiation.cc:399] Waiting for connection header
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.988851 (+ 48us) server_negotiation.cc:407] Connection header received
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.989024 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.989064 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.989337 (+ 273us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.989553 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.992408 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46850: BlockingRecv error: recv got EOF from 127.0.0.1:46850 (error 108)
23:40:40.993 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1013,"thread_start_us":723,"threads_started":1}
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:40.993829 8015 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.988676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56528 (local address 127.7.24.126:39219)
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.989719 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.989733 (+ 14us) server_negotiation.cc:399] Waiting for connection header
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.989768 (+ 35us) server_negotiation.cc:407] Connection header received
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.989869 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.989885 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.990038 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.990519 (+ 481us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:40.992703 (+ 2184us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56528: BlockingRecv error: recv got EOF from 127.0.0.1:56528 (error 108)
23:40:40.994 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":508,"threads_started":1}
23:40:42.165 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:42.166 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:42.166 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:42.166 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:42.170 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:42.171 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:42.171 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:42.171 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:42.171 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:42.171 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:42.172 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:42.172894 8020 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.168067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.7.24.126:39219)
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.169507 (+ 1440us) server_negotiation.cc:206] Beginning negotiation
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.169531 (+ 24us) server_negotiation.cc:399] Waiting for connection header
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.169582 (+ 51us) server_negotiation.cc:407] Connection header received
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.169724 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.169751 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.169974 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.171149 (+ 1175us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.171819 (+ 670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56542: BlockingRecv error: recv got EOF from 127.0.0.1:56542 (error 108)
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":673,"threads_started":1}
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:42.173504 8019 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.167679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46854 (local address 127.7.24.125:35285)
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.169920 (+ 2241us) server_negotiation.cc:206] Beginning negotiation
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.169934 (+ 14us) server_negotiation.cc:399] Waiting for connection header
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.169970 (+ 36us) server_negotiation.cc:407] Connection header received
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.170076 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:42.173 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.170121 (+ 45us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:42.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.170574 (+ 453us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:42.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.170756 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:42.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:42.172381 (+ 1625us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46854: BlockingRecv error: recv got EOF from 127.0.0.1:46854 (error 108)
23:40:42.174 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1941,"thread_start_us":1659,"threads_started":1}
23:40:44.265 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:44.266 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:44.266 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:44.268 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:44.273 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:44.273 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:44.273 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:44.273 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:44.274 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:44.274 [WARN - kudu-nio-0] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:44.275 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:44.276683 8025 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.268555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.7.24.125:35285)
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.271901 (+ 3346us) server_negotiation.cc:206] Beginning negotiation
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.271932 (+ 31us) server_negotiation.cc:399] Waiting for connection header
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.271988 (+ 56us) server_negotiation.cc:407] Connection header received
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.272167 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.272199 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.272455 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.274445 (+ 1990us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.275293 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46862: BlockingRecv error: recv got EOF from 127.0.0.1:46862 (error 108)
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2832,"thread_start_us":2511,"threads_started":1}
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:44.276765 8026 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.269195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.7.24.126:39219)
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.272386 (+ 3191us) server_negotiation.cc:206] Beginning negotiation
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.272410 (+ 24us) server_negotiation.cc:399] Waiting for connection header
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.272460 (+ 50us) server_negotiation.cc:407] Connection header received
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.272621 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.272646 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.273010 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.274190 (+ 1180us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:44.275492 (+ 1302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56546: BlockingRecv error: recv got EOF from 127.0.0.1:56546 (error 108)
23:40:44.277 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":2721,"thread_start_us":2401,"threads_started":1}
23:40:45.618 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Aug 10 23:40:45 dist-test-slave-9jjr krb5kdc[7277](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786405245, etypes {rep=17 tkt=17 ses=17}, kudu/127.7.24.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
23:40:45.619 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:45.619474 7643 init.cc:295] Successfully reacquired a new kerberos TGT
23:40:47.545 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.125:35285(127.7.24.125:35285)] connecting to peer
23:40:47.546 [DEBUG - pool-48-thread-1] (Connection.java:748) [peer master-127.7.24.126:39219(127.7.24.126:39219)] connecting to peer
23:40:47.549 [DEBUG - kudu-nio-0] (Connection.java:757) [peer master-127.7.24.126:39219(127.7.24.126:39219)] Successfully connected to peer
23:40:47.549 [DEBUG - kudu-nio-1] (Connection.java:757) [peer master-127.7.24.125:35285(127.7.24.125:35285)] Successfully connected to peer
23:40:47.553 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.126:39219(127.7.24.126:39219)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:47.553 [INFO - kudu-nio-0] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.126:39219: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:47.553 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.126:39219(127.7.24.126:39219)] handling channelInactive
23:40:47.553 [DEBUG - kudu-nio-1] (Connection.java:709) [peer master-127.7.24.125:35285(127.7.24.125:35285)] cleaning up while in state NEGOTIATING due to: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:47.553 [INFO - kudu-nio-1] (ConnectToCluster.java:385) Unable to connect to master 127.7.24.125:35285: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
23:40:47.554 [WARN - kudu-nio-1] (ConnectToCluster.java:305) Unable to find the leader master 127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219; will retry
23:40:47.555 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:1887) Cannot continue with RPC because of: Timed out: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=23, TimeoutTracker(timeout=30000, elapsed=27654), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [66ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [118ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [126ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [136ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [141ms] refreshing cache from master, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [149ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [154ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [159ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [165ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [180ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [186ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [192ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [196ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [196ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [201ms] refreshing cache from master, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [202ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [206ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [210ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [238ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [239ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [260ms] refreshing cache from master, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [267ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [267ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [278ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [278ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [300ms] refreshing cache from master, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [315ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [317ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [320ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [320ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [341ms] refreshing cache from master, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [351ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [420ms] refreshing cache from master, [420ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [425ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [428ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [429ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [430ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [680ms] refreshing cache from master, [680ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [685ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [686ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [690ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [690ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [780ms] refreshing cache from master, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [782ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [785ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [785ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [786ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [787ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1160ms] refreshing cache from master, [1161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [1161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [1162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1166ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1167ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1620ms] refreshing cache from master, [1621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [1621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [1622ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [1625ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [1626ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1627ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1628ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [5220ms] refreshing cache from master, [5220ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [5221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [5221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [5225ms] too many traces: truncated at 100 traces, deferred=Deferred@1739614207(state=PENDING, result=null, callback=(continuation of Deferred@699091687 after retry RPC after error@1214679178) -> (continuation of Deferred@1722915596 after retry RPC after error@39491514) -> (continuation of Deferred@1901398372 after retry RPC after error@218889500) -> (continuation of Deferred@1728605770 after retry RPC after error@834257494) -> (continuation of Deferred@1753150252 after retry RPC after error@986648610) -> (continuation of Deferred@512836367 after retry RPC after error@1640335865) -> (continuation of Deferred@2117988734 after retry RPC after error@1018278043) -> (continuation of Deferred@673977657 after retry RPC after error@1709951876) -> (continuation of Deferred@1959551046 after retry RPC after error@1170748204) -> (continuation of Deferred@226845553 after retry RPC after error@914960046) -> (continuation of Deferred@558937705 after retry RPC after error@621129788) -> (continuation of Deferred@54378758 after retry RPC after error@2092856209) -> (continuation of Deferred@1492742545 after retry RPC after error@1349889762) -> (continuation of Deferred@1026383899 after retry RPC after error@838909652) -> (continuation of Deferred@568734972 after retry RPC after error@1545182364) -> (continuation of Deferred@174195388 after retry RPC after error@1034152762) -> (continuation of Deferred@762476966 after retry RPC after error@1025502696) -> (continuation of Deferred@661815061 after retry RPC after error@95075066) -> (continuation of Deferred@992693408 after retry RPC after error@2040108672) -> (continuation of Deferred@1302844529 after retry RPC after error@495635029) -> (continuation of Deferred@2091240200 after retry RPC after error@1237155722) -> (continuation of Deferred@986328720 after retry RPC after error@1901695998), errback=(continuation of Deferred@699091687 after retry RPC after error@1214679178) -> (continuation of Deferred@1722915596 after retry RPC after error@39491514) -> (continuation of Deferred@1901398372 after retry RPC after error@218889500) -> (continuation of Deferred@1728605770 after retry RPC after error@834257494) -> (continuation of Deferred@1753150252 after retry RPC after error@986648610) -> (continuation of Deferred@512836367 after retry RPC after error@1640335865) -> (continuation of Deferred@2117988734 after retry RPC after error@1018278043) -> (continuation of Deferred@673977657 after retry RPC after error@1709951876) -> (continuation of Deferred@1959551046 after retry RPC after error@1170748204) -> (continuation of Deferred@226845553 after retry RPC after error@914960046) -> (continuation of Deferred@558937705 after retry RPC after error@621129788) -> (continuation of Deferred@54378758 after retry RPC after error@2092856209) -> (continuation of Deferred@1492742545 after retry RPC after error@1349889762) -> (continuation of Deferred@1026383899 after retry RPC after error@838909652) -> (continuation of Deferred@568734972 after retry RPC after error@1545182364) -> (continuation of Deferred@174195388 after retry RPC after error@1034152762) -> (continuation of Deferred@762476966 after retry RPC after error@1025502696) -> (continuation of Deferred@661815061 after retry RPC after error@95075066) -> (continuation of Deferred@992693408 after retry RPC after error@2040108672) -> (continuation of Deferred@1302844529 after retry RPC after error@495635029) -> (continuation of Deferred@2091240200 after retry RPC after error@1237155722) -> (continuation of Deferred@986328720 after retry RPC after error@1901695998)))
23:40:47.556 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:47.556425 8030 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.550428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48660 (local address 127.7.24.126:39219)
23:40:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.551746 (+ 1318us) server_negotiation.cc:206] Beginning negotiation
23:40:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.551767 (+ 21us) server_negotiation.cc:399] Waiting for connection header
23:40:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.551815 (+ 48us) server_negotiation.cc:407] Connection header received
23:40:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.551952 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.551989 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.552212 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.553797 (+ 1585us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.554495 (+ 698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48660: BlockingRecv error: recv got EOF from 127.0.0.1:48660 (error 108)
23:40:47.559 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":630,"threads_started":1}
23:40:47.561 [DEBUG - main] (Connection.java:575) [peer master-127.7.24.125:35285(127.7.24.125:35285)] disconnecting while in state TERMINATED
23:40:47.561 [DEBUG - main] (Connection.java:575) [peer master-127.7.24.126:39219(127.7.24.126:39219)] disconnecting while in state TERMINATED
23:40:47.561 [DEBUG - main] (Connection.java:575) [peer master-127.7.24.124:41745(127.7.24.124:41745)] disconnecting while in state READY
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:47.562204 8031 negotiation.cc:337] Failed RPC negotiation. Trace:
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.549949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33138 (local address 127.7.24.125:35285)
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.552387 (+ 2438us) server_negotiation.cc:206] Beginning negotiation
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.552421 (+ 34us) server_negotiation.cc:399] Waiting for connection header
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.552483 (+ 62us) server_negotiation.cc:407] Connection header received
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.552689 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.552714 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.552964 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.557934 (+ 4970us) server_negotiation.cc:226] Negotiated authn=SASL
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 0810 23:40:47.560778 (+ 2844us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33138: BlockingRecv error: recv got EOF from 127.0.0.1:33138 (error 108)
23:40:47.562 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) Metrics: {"server-negotiator.queue_time_us":1935,"thread_start_us":3445,"threads_started":1}
23:40:47.563 [DEBUG - kudu-nio-0] (Connection.java:239) [peer master-127.7.24.124:41745(127.7.24.124:41745)] handling channelInactive
23:40:47.564 [DEBUG - kudu-nio-0] (Connection.java:709) [peer master-127.7.24.124:41745(127.7.24.124:41745)] cleaning up while in state READY due to: connection closed
23:40:47.565 [DEBUG - kudu-nio-1] (AsyncKuduClient.java:2704) Releasing all remaining resources
23:40:47.565 [DEBUG - kudu-nio-1] (Connection.java:239) [peer master-127.7.24.125:35285(127.7.24.125:35285)] handling channelInactive
23:40:47.567 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:47.567294 7265 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 7814
23:40:47.606 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:47.606585 7265 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 7711
23:40:47.643 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:47.643229 7265 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 7630
23:40:48.826 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-10T23:40:48Z chronyd exiting
23:40:49.873 [DEBUG - main] (AsyncKuduClient.java:2704) Releasing all remaining resources
23:40:49.875 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:49.875358 6456 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 6768
23:40:49.909 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:49.909471 6456 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 6954
23:40:49.945 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:49.945427 6456 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 7106
23:40:49.977 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:49.977286 6456 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 6469
23:40:50.019 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:50.019030 6456 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 6545
23:40:50.059 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:50.059230 6661 connection.cc:535] client connection to 127.6.78.61:36805 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
23:40:50.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) I20260810 23:40:50.059929 6456 external_mini_cluster.cc:1595] Killing /tmp/dist-test-tasksngkcS/build/tsan/bin/kudu with pid 6627
23:40:50.060 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) W20260810 23:40:50.059834 6661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
23:40:51.217 [INFO - cluster stderr printer] (MiniKuduCluster.java:751) 2026-08-10T23:40:51Z chronyd exiting
23:40:52.265 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: failed attempt 1
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=23, TimeoutTracker(timeout=30000, elapsed=27654), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [66ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [118ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [126ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [136ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [141ms] refreshing cache from master, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [149ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [154ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [159ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [165ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [180ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [186ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [192ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [196ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [196ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [201ms] refreshing cache from master, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [202ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [206ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [210ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [238ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [239ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [260ms] refreshing cache from master, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [267ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [267ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [278ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [278ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [300ms] refreshing cache from master, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [315ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [317ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [320ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [320ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [341ms] refreshing cache from master, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [351ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [420ms] refreshing cache from master, [420ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [425ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [428ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [429ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [430ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [680ms] refreshing cache from master, [680ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [685ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [686ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [690ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [690ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [780ms] refreshing cache from master, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [782ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [785ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [785ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [786ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [787ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1160ms] refreshing cache from master, [1161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [1161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [1162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1166ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1167ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1620ms] refreshing cache from master, [1621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [1621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [1622ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [1625ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [1626ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1627ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1628ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [5220ms] refreshing cache from master, [5220ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [5221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [5221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [5225ms] too many traces: truncated at 100 traces, deferred=Deferred@1739614207(state=PENDING, result=null, callback=(continuation of Deferred@699091687 after retry RPC after error@1214679178) -> (continuation of Deferred@1722915596 after retry RPC after error@39491514) -> (continuation of Deferred@1901398372 after retry RPC after error@218889500) -> (continuation of Deferred@1728605770 after retry RPC after error@834257494) -> (continuation of Deferred@1753150252 after retry RPC after error@986648610) -> (continuation of Deferred@512836367 after retry RPC after error@1640335865) -> (continuation of Deferred@2117988734 after retry RPC after error@1018278043) -> (continuation of Deferred@673977657 after retry RPC after error@1709951876) -> (continuation of Deferred@1959551046 after retry RPC after error@1170748204) -> (continuation of Deferred@226845553 after retry RPC after error@914960046) -> (continuation of Deferred@558937705 after retry RPC after error@621129788) -> (continuation of Deferred@54378758 after retry RPC after error@2092856209) -> (continuation of Deferred@1492742545 after retry RPC after error@1349889762) -> (continuation of Deferred@1026383899 after retry RPC after error@838909652) -> (continuation of Deferred@568734972 after retry RPC after error@1545182364) -> (continuation of Deferred@174195388 after retry RPC after error@1034152762) -> (continuation of Deferred@762476966 after retry RPC after error@1025502696) -> (continuation of Deferred@661815061 after retry RPC after error@95075066) -> (continuation of Deferred@992693408 after retry RPC after error@2040108672) -> (continuation of Deferred@1302844529 after retry RPC after error@495635029) -> (continuation of Deferred@2091240200 after retry RPC after error@1237155722) -> (continuation of Deferred@986328720 after retry RPC after error@1901695998), errback=(continuation of Deferred@699091687 after retry RPC after error@1214679178) -> (continuation of Deferred@1722915596 after retry RPC after error@39491514) -> (continuation of Deferred@1901398372 after retry RPC after error@218889500) -> (continuation of Deferred@1728605770 after retry RPC after error@834257494) -> (continuation of Deferred@1753150252 after retry RPC after error@986648610) -> (continuation of Deferred@512836367 after retry RPC after error@1640335865) -> (continuation of Deferred@2117988734 after retry RPC after error@1018278043) -> (continuation of Deferred@673977657 after retry RPC after error@1709951876) -> (continuation of Deferred@1959551046 after retry RPC after error@1170748204) -> (continuation of Deferred@226845553 after retry RPC after error@914960046) -> (continuation of Deferred@558937705 after retry RPC after error@621129788) -> (continuation of Deferred@54378758 after retry RPC after error@2092856209) -> (continuation of Deferred@1492742545 after retry RPC after error@1349889762) -> (continuation of Deferred@1026383899 after retry RPC after error@838909652) -> (continuation of Deferred@568734972 after retry RPC after error@1545182364) -> (continuation of Deferred@174195388 after retry RPC after error@1034152762) -> (continuation of Deferred@762476966 after retry RPC after error@1025502696) -> (continuation of Deferred@661815061 after retry RPC after error@95075066) -> (continuation of Deferred@992693408 after retry RPC after error@2040108672) -> (continuation of Deferred@1302844529 after retry RPC after error@495635029) -> (continuation of Deferred@2091240200 after retry RPC after error@1237155722) -> (continuation of Deferred@986328720 after retry RPC after error@1901695998)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) ~[main/:?]
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564) ~[main/:?]
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263) ~[main/:?]
at org.apache.kudu.client.TestSecurity$3.get(TestSecurity.java:386) ~[test/:?]
at org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40) ~[kudu-test-utils-1.18.0-SNAPSHOT-unshaded.jar:1.18.0-SNAPSHOT]
at org.apache.kudu.client.TestSecurity.assertEventualAuthenticationFailure(TestSecurity.java:378) ~[test/:?]
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:366) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565) ~[main/:?]
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94) ~[main/:?]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=23, TimeoutTracker(timeout=30000, elapsed=27654), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [66ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [118ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [126ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [136ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [141ms] refreshing cache from master, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [149ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [154ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [159ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [165ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [180ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [186ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [192ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [196ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [196ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [201ms] refreshing cache from master, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [202ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [206ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [210ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [238ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [239ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [260ms] refreshing cache from master, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [267ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [267ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [278ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [278ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [300ms] refreshing cache from master, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [315ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [317ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [320ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [320ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [341ms] refreshing cache from master, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [351ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [420ms] refreshing cache from master, [420ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [425ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [428ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [429ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [430ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [680ms] refreshing cache from master, [680ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [685ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [686ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [690ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [690ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [780ms] refreshing cache from master, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [782ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [785ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [785ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [786ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [787ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1160ms] refreshing cache from master, [1161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [1161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [1162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1166ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1167ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1620ms] refreshing cache from master, [1621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [1621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [1622ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [1625ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [1626ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1627ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1628ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [5220ms] refreshing cache from master, [5220ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [5221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [5221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [5225ms] too many traces: truncated at 100 traces, deferred=Deferred@1739614207(state=PENDING, result=null, callback=(continuation of Deferred@699091687 after retry RPC after error@1214679178) -> (continuation of Deferred@1722915596 after retry RPC after error@39491514) -> (continuation of Deferred@1901398372 after retry RPC after error@218889500) -> (continuation of Deferred@1728605770 after retry RPC after error@834257494) -> (continuation of Deferred@1753150252 after retry RPC after error@986648610) -> (continuation of Deferred@512836367 after retry RPC after error@1640335865) -> (continuation of Deferred@2117988734 after retry RPC after error@1018278043) -> (continuation of Deferred@673977657 after retry RPC after error@1709951876) -> (continuation of Deferred@1959551046 after retry RPC after error@1170748204) -> (continuation of Deferred@226845553 after retry RPC after error@914960046) -> (continuation of Deferred@558937705 after retry RPC after error@621129788) -> (continuation of Deferred@54378758 after retry RPC after error@2092856209) -> (continuation of Deferred@1492742545 after retry RPC after error@1349889762) -> (continuation of Deferred@1026383899 after retry RPC after error@838909652) -> (continuation of Deferred@568734972 after retry RPC after error@1545182364) -> (continuation of Deferred@174195388 after retry RPC after error@1034152762) -> (continuation of Deferred@762476966 after retry RPC after error@1025502696) -> (continuation of Deferred@661815061 after retry RPC after error@95075066) -> (continuation of Deferred@992693408 after retry RPC after error@2040108672) -> (continuation of Deferred@1302844529 after retry RPC after error@495635029) -> (continuation of Deferred@2091240200 after retry RPC after error@1237155722) -> (continuation of Deferred@986328720 after retry RPC after error@1901695998), errback=(continuation of Deferred@699091687 after retry RPC after error@1214679178) -> (continuation of Deferred@1722915596 after retry RPC after error@39491514) -> (continuation of Deferred@1901398372 after retry RPC after error@218889500) -> (continuation of Deferred@1728605770 after retry RPC after error@834257494) -> (continuation of Deferred@1753150252 after retry RPC after error@986648610) -> (continuation of Deferred@512836367 after retry RPC after error@1640335865) -> (continuation of Deferred@2117988734 after retry RPC after error@1018278043) -> (continuation of Deferred@673977657 after retry RPC after error@1709951876) -> (continuation of Deferred@1959551046 after retry RPC after error@1170748204) -> (continuation of Deferred@226845553 after retry RPC after error@914960046) -> (continuation of Deferred@558937705 after retry RPC after error@621129788) -> (continuation of Deferred@54378758 after retry RPC after error@2092856209) -> (continuation of Deferred@1492742545 after retry RPC after error@1349889762) -> (continuation of Deferred@1026383899 after retry RPC after error@838909652) -> (continuation of Deferred@568734972 after retry RPC after error@1545182364) -> (continuation of Deferred@174195388 after retry RPC after error@1034152762) -> (continuation of Deferred@762476966 after retry RPC after error@1025502696) -> (continuation of Deferred@661815061 after retry RPC after error@95075066) -> (continuation of Deferred@992693408 after retry RPC after error@2040108672) -> (continuation of Deferred@1302844529 after retry RPC after error@495635029) -> (continuation of Deferred@2091240200 after retry RPC after error@1237155722) -> (continuation of Deferred@986328720 after retry RPC after error@1901695998)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564)
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263)
at org.apache.kudu.client.TestSecurity$3.get(TestSecurity.java:386)
at org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
at org.apache.kudu.client.TestSecurity.assertEventualAuthenticationFailure(TestSecurity.java:378)
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889)
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337)
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565)
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
... 30 more
23:40:52.804 [ERROR - main] (RetryRule.java:253) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: giving up after 1 attempts
23:40:52.804 [ERROR - main] (RetryRule.java:206) org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient: failed attempt 1
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=23, TimeoutTracker(timeout=30000, elapsed=27654), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [66ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [118ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [126ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [136ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [141ms] refreshing cache from master, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [149ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [154ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [159ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [165ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [180ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [186ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [192ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [196ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [196ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [201ms] refreshing cache from master, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [202ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [206ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [210ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [238ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [239ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [260ms] refreshing cache from master, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [267ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [267ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [278ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [278ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [300ms] refreshing cache from master, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [315ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [317ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [320ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [320ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [341ms] refreshing cache from master, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [351ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [420ms] refreshing cache from master, [420ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [425ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [428ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [429ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [430ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [680ms] refreshing cache from master, [680ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [685ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [686ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [690ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [690ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [780ms] refreshing cache from master, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [782ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [785ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [785ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [786ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [787ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1160ms] refreshing cache from master, [1161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [1161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [1162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1166ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1167ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1620ms] refreshing cache from master, [1621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [1621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [1622ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [1625ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [1626ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1627ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1628ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [5220ms] refreshing cache from master, [5220ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [5221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [5221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [5225ms] too many traces: truncated at 100 traces, deferred=Deferred@1739614207(state=PENDING, result=null, callback=(continuation of Deferred@699091687 after retry RPC after error@1214679178) -> (continuation of Deferred@1722915596 after retry RPC after error@39491514) -> (continuation of Deferred@1901398372 after retry RPC after error@218889500) -> (continuation of Deferred@1728605770 after retry RPC after error@834257494) -> (continuation of Deferred@1753150252 after retry RPC after error@986648610) -> (continuation of Deferred@512836367 after retry RPC after error@1640335865) -> (continuation of Deferred@2117988734 after retry RPC after error@1018278043) -> (continuation of Deferred@673977657 after retry RPC after error@1709951876) -> (continuation of Deferred@1959551046 after retry RPC after error@1170748204) -> (continuation of Deferred@226845553 after retry RPC after error@914960046) -> (continuation of Deferred@558937705 after retry RPC after error@621129788) -> (continuation of Deferred@54378758 after retry RPC after error@2092856209) -> (continuation of Deferred@1492742545 after retry RPC after error@1349889762) -> (continuation of Deferred@1026383899 after retry RPC after error@838909652) -> (continuation of Deferred@568734972 after retry RPC after error@1545182364) -> (continuation of Deferred@174195388 after retry RPC after error@1034152762) -> (continuation of Deferred@762476966 after retry RPC after error@1025502696) -> (continuation of Deferred@661815061 after retry RPC after error@95075066) -> (continuation of Deferred@992693408 after retry RPC after error@2040108672) -> (continuation of Deferred@1302844529 after retry RPC after error@495635029) -> (continuation of Deferred@2091240200 after retry RPC after error@1237155722) -> (continuation of Deferred@986328720 after retry RPC after error@1901695998), errback=(continuation of Deferred@699091687 after retry RPC after error@1214679178) -> (continuation of Deferred@1722915596 after retry RPC after error@39491514) -> (continuation of Deferred@1901398372 after retry RPC after error@218889500) -> (continuation of Deferred@1728605770 after retry RPC after error@834257494) -> (continuation of Deferred@1753150252 after retry RPC after error@986648610) -> (continuation of Deferred@512836367 after retry RPC after error@1640335865) -> (continuation of Deferred@2117988734 after retry RPC after error@1018278043) -> (continuation of Deferred@673977657 after retry RPC after error@1709951876) -> (continuation of Deferred@1959551046 after retry RPC after error@1170748204) -> (continuation of Deferred@226845553 after retry RPC after error@914960046) -> (continuation of Deferred@558937705 after retry RPC after error@621129788) -> (continuation of Deferred@54378758 after retry RPC after error@2092856209) -> (continuation of Deferred@1492742545 after retry RPC after error@1349889762) -> (continuation of Deferred@1026383899 after retry RPC after error@838909652) -> (continuation of Deferred@568734972 after retry RPC after error@1545182364) -> (continuation of Deferred@174195388 after retry RPC after error@1034152762) -> (continuation of Deferred@762476966 after retry RPC after error@1025502696) -> (continuation of Deferred@661815061 after retry RPC after error@95075066) -> (continuation of Deferred@992693408 after retry RPC after error@2040108672) -> (continuation of Deferred@1302844529 after retry RPC after error@495635029) -> (continuation of Deferred@2091240200 after retry RPC after error@1237155722) -> (continuation of Deferred@986328720 after retry RPC after error@1901695998)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110) ~[main/:?]
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564) ~[main/:?]
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263) ~[main/:?]
at org.apache.kudu.client.TestSecurity$3.get(TestSecurity.java:386) ~[test/:?]
at org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40) ~[kudu-test-utils-1.18.0-SNAPSHOT-unshaded.jar:1.18.0-SNAPSHOT]
at org.apache.kudu.client.TestSecurity.assertEventualAuthenticationFailure(TestSecurity.java:378) ~[test/:?]
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:366) ~[test/:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_362]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_362]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_362]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_362]
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) ~[junit-4.13.2.jar:4.13.2]
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299) ~[junit-4.13.2.jar:4.13.2]
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293) ~[junit-4.13.2.jar:4.13.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520) ~[main/:?]
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_362]
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387) ~[main/:?]
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376) ~[main/:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259) ~[async-1.4.1.jar:?]
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002) ~[async-1.4.1.jar:?]
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277) ~[main/:?]
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331) ~[main/:?]
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256) ~[main/:?]
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158) ~[main/:?]
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154) ~[main/:?]
at org.apache.kudu.client.Connection.cleanup(Connection.java:737) ~[main/:?]
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455) ~[main/:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565) ~[main/:?]
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367) ~[main/:?]
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94) ~[main/:?]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) ~[netty-handler-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[netty-transport-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.94.Final.jar:4.1.94.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_362]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_362]
... 1 more
org.apache.kudu.client.NonRecoverableException: cannot complete before timeout: KuduRpc(method=ListTabletServers, tablet=null, attempt=23, TimeoutTracker(timeout=30000, elapsed=27654), Traces: [0ms] refreshing cache from master, [0ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [2ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [3ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [66ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [118ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [126ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Configuration error: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [136ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT),org.apache.kudu.client.NonRecoverableException: Server requires Kerberos, but this client is not authenticated (missing or expired TGT), [141ms] refreshing cache from master, [142ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [147ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [149ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [154ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [159ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [165ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [180ms] refreshing cache from master, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [181ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [186ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [192ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [196ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [196ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [201ms] refreshing cache from master, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [201ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [202ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [206ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [210ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [238ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [239ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [260ms] refreshing cache from master, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [261ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [262ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [267ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [267ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [278ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [278ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [300ms] refreshing cache from master, [300ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [301ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [315ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [317ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [320ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [320ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [341ms] refreshing cache from master, [341ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [342ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [349ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [351ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [420ms] refreshing cache from master, [420ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [421ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [425ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [428ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [429ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [430ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [680ms] refreshing cache from master, [680ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [681ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [685ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [686ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [690ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [690ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [780ms] refreshing cache from master, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [781ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [782ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [785ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [785ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [786ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [787ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1160ms] refreshing cache from master, [1161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [1161ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [1162ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [1165ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1166ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1167ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1620ms] refreshing cache from master, [1621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [1621ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [1622ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [1625ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.124:41745: OK, [1626ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.126:39219: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1627ms] Sub RPC ConnectToMaster: received response from server master-127.7.24.125:35285: Not authorized: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [1628ms] delaying RPC due to: Service unavailable: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available, [5220ms] refreshing cache from master, [5220ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.124:41745, [5221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.125:35285, [5221ms] Sub RPC ConnectToMaster: sending RPC to server master-127.7.24.126:39219, [5225ms] too many traces: truncated at 100 traces, deferred=Deferred@1739614207(state=PENDING, result=null, callback=(continuation of Deferred@699091687 after retry RPC after error@1214679178) -> (continuation of Deferred@1722915596 after retry RPC after error@39491514) -> (continuation of Deferred@1901398372 after retry RPC after error@218889500) -> (continuation of Deferred@1728605770 after retry RPC after error@834257494) -> (continuation of Deferred@1753150252 after retry RPC after error@986648610) -> (continuation of Deferred@512836367 after retry RPC after error@1640335865) -> (continuation of Deferred@2117988734 after retry RPC after error@1018278043) -> (continuation of Deferred@673977657 after retry RPC after error@1709951876) -> (continuation of Deferred@1959551046 after retry RPC after error@1170748204) -> (continuation of Deferred@226845553 after retry RPC after error@914960046) -> (continuation of Deferred@558937705 after retry RPC after error@621129788) -> (continuation of Deferred@54378758 after retry RPC after error@2092856209) -> (continuation of Deferred@1492742545 after retry RPC after error@1349889762) -> (continuation of Deferred@1026383899 after retry RPC after error@838909652) -> (continuation of Deferred@568734972 after retry RPC after error@1545182364) -> (continuation of Deferred@174195388 after retry RPC after error@1034152762) -> (continuation of Deferred@762476966 after retry RPC after error@1025502696) -> (continuation of Deferred@661815061 after retry RPC after error@95075066) -> (continuation of Deferred@992693408 after retry RPC after error@2040108672) -> (continuation of Deferred@1302844529 after retry RPC after error@495635029) -> (continuation of Deferred@2091240200 after retry RPC after error@1237155722) -> (continuation of Deferred@986328720 after retry RPC after error@1901695998), errback=(continuation of Deferred@699091687 after retry RPC after error@1214679178) -> (continuation of Deferred@1722915596 after retry RPC after error@39491514) -> (continuation of Deferred@1901398372 after retry RPC after error@218889500) -> (continuation of Deferred@1728605770 after retry RPC after error@834257494) -> (continuation of Deferred@1753150252 after retry RPC after error@986648610) -> (continuation of Deferred@512836367 after retry RPC after error@1640335865) -> (continuation of Deferred@2117988734 after retry RPC after error@1018278043) -> (continuation of Deferred@673977657 after retry RPC after error@1709951876) -> (continuation of Deferred@1959551046 after retry RPC after error@1170748204) -> (continuation of Deferred@226845553 after retry RPC after error@914960046) -> (continuation of Deferred@558937705 after retry RPC after error@621129788) -> (continuation of Deferred@54378758 after retry RPC after error@2092856209) -> (continuation of Deferred@1492742545 after retry RPC after error@1349889762) -> (continuation of Deferred@1026383899 after retry RPC after error@838909652) -> (continuation of Deferred@568734972 after retry RPC after error@1545182364) -> (continuation of Deferred@174195388 after retry RPC after error@1034152762) -> (continuation of Deferred@762476966 after retry RPC after error@1025502696) -> (continuation of Deferred@661815061 after retry RPC after error@95075066) -> (continuation of Deferred@992693408 after retry RPC after error@2040108672) -> (continuation of Deferred@1302844529 after retry RPC after error@495635029) -> (continuation of Deferred@2091240200 after retry RPC after error@1237155722) -> (continuation of Deferred@986328720 after retry RPC after error@1901695998)))
at org.apache.kudu.client.KuduException.transformException(KuduException.java:110)
at org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:564)
at org.apache.kudu.client.KuduClient.listTabletServers(KuduClient.java:263)
at org.apache.kudu.client.TestSecurity$3.get(TestSecurity.java:386)
at org.apache.kudu.test.junit.AssertHelpers.assertEventuallyTrue(AssertHelpers.java:40)
at org.apache.kudu.client.TestSecurity.assertEventualAuthenticationFailure(TestSecurity.java:378)
at org.apache.kudu.client.TestSecurity.testDoNotSwitchPrincipalsInExistingClient(TestSecurity.java:366)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:750)
Suppressed: org.apache.kudu.client.KuduException$OriginalException: Original asynchronous stack trace
at org.apache.kudu.client.AsyncKuduClient.tooManyAttemptsOrTimeout(AsyncKuduClient.java:1889)
at org.apache.kudu.client.AsyncKuduClient.delayedSendRpcToTablet(AsyncKuduClient.java:2337)
at org.apache.kudu.client.AsyncKuduClient.access$1500(AsyncKuduClient.java:299)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1520)
at org.apache.kudu.client.AsyncKuduClient$RetryRpcErrback.call(AsyncKuduClient.java:1510)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:313)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NoLeaderFoundException: Master config (127.7.24.124:41745,127.7.24.125:35285,127.7.24.126:39219) has no leader. Exceptions received: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available,org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.ConnectToCluster.incrementCountAndCheckExhausted(ConnectToCluster.java:311)
at org.apache.kudu.client.ConnectToCluster.access$000(ConnectToCluster.java:51)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:387)
at org.apache.kudu.client.ConnectToCluster$ConnectToMasterErrCB.call(ConnectToCluster.java:376)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1280)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.handleContinuation(Deferred.java:1315)
at com.stumbleupon.async.Deferred.doCall(Deferred.java:1286)
at com.stumbleupon.async.Deferred.runCallbacks(Deferred.java:1259)
at com.stumbleupon.async.Deferred.callback(Deferred.java:1002)
at org.apache.kudu.client.KuduRpc.handleCallback(KuduRpc.java:277)
at org.apache.kudu.client.KuduRpc.errback(KuduRpc.java:331)
at org.apache.kudu.client.RpcProxy.responseReceived(RpcProxy.java:256)
at org.apache.kudu.client.RpcProxy.access$000(RpcProxy.java:64)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:158)
at org.apache.kudu.client.RpcProxy$1.call(RpcProxy.java:154)
at org.apache.kudu.client.Connection.cleanup(Connection.java:737)
at org.apache.kudu.client.Connection.exceptionCaught(Connection.java:455)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:325)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:317)
at io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:143)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:346)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:447)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
... 1 more
Caused by: org.apache.kudu.client.NonRecoverableException: server requires authentication, but client does not have Kerberos credentials (tgt). Authentication tokens were not used because no token is available
at org.apache.kudu.client.Negotiator.chooseAndInitializeSaslMech(Negotiator.java:565)
at org.apache.kudu.client.Negotiator.handleNegotiateResponse(Negotiator.java:433)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:367)
at org.apache.kudu.client.Negotiator.channelRead0(Negotiator.java:94)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
... 30 more