Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250809 02:04:57.848335 26343 test_util.cc:276] Using random seed: 504034697
W20250809 02:04:59.069952 26343 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.176s user 0.462s sys 0.711s
W20250809 02:04:59.070434 26343 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.177s user 0.462s sys 0.711s
I20250809 02:04:59.073302 26343 ts_itest-base.cc:115] Starting cluster with:
I20250809 02:04:59.073552 26343 ts_itest-base.cc:116] --------------
I20250809 02:04:59.073769 26343 ts_itest-base.cc:117] 4 tablet servers
I20250809 02:04:59.073982 26343 ts_itest-base.cc:118] 3 replicas per TS
I20250809 02:04:59.074185 26343 ts_itest-base.cc:119] --------------
2025-08-09T02:04:59Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T02:04:59Z Disabled control of system clock
I20250809 02:04:59.117857 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:44687
--webserver_interface=127.25.185.254
--webserver_port=0
--builtin_ntp_servers=127.25.185.212:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:44687 with env {}
W20250809 02:04:59.423766 26357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:04:59.424381 26357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:04:59.424865 26357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:04:59.455828 26357 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:04:59.456164 26357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:04:59.456434 26357 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:04:59.456703 26357 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:04:59.493674 26357 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:44687
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:44687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:04:59.495051 26357 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:04:59.496583 26357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:04:59.508896 26363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:04:59.510206 26364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:00.715759 26365 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1200 milliseconds
W20250809 02:05:00.716459 26357 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.207s user 0.401s sys 0.801s
W20250809 02:05:00.716710 26357 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.208s user 0.401s sys 0.801s
W20250809 02:05:00.717121 26366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:00.717202 26357 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:00.718430 26357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:00.721433 26357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:00.722805 26357 hybrid_clock.cc:648] HybridClock initialized: now 1754705100722753 us; error 67 us; skew 500 ppm
I20250809 02:05:00.723584 26357 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:00.732264 26357 webserver.cc:489] Webserver started at http://127.25.185.254:38791/ using document root <none> and password file <none>
I20250809 02:05:00.733192 26357 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:00.733415 26357 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:00.733840 26357 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:00.738694 26357 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2f155abab92e41fe90c86d52b9b170ba"
format_stamp: "Formatted at 2025-08-09 02:05:00 on dist-test-slave-24ff"
I20250809 02:05:00.739836 26357 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2f155abab92e41fe90c86d52b9b170ba"
format_stamp: "Formatted at 2025-08-09 02:05:00 on dist-test-slave-24ff"
I20250809 02:05:00.748353 26357 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20250809 02:05:00.754343 26373 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:00.755713 26357 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250809 02:05:00.756063 26357 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2f155abab92e41fe90c86d52b9b170ba"
format_stamp: "Formatted at 2025-08-09 02:05:00 on dist-test-slave-24ff"
I20250809 02:05:00.756410 26357 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:00.830660 26357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:00.832655 26357 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:00.833108 26357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:00.912652 26357 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:44687
I20250809 02:05:00.912701 26424 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:44687 every 8 connection(s)
I20250809 02:05:00.915462 26357 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 02:05:00.921159 26425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:00.923285 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 26357
I20250809 02:05:00.923679 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 02:05:00.941606 26425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba: Bootstrap starting.
I20250809 02:05:00.947341 26425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:00.949450 26425 log.cc:826] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:00.954869 26425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba: No bootstrap required, opened a new log
I20250809 02:05:00.974139 26425 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } }
I20250809 02:05:00.974820 26425 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:00.975049 26425 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f155abab92e41fe90c86d52b9b170ba, State: Initialized, Role: FOLLOWER
I20250809 02:05:00.975736 26425 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } }
I20250809 02:05:00.976253 26425 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:05:00.976501 26425 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:05:00.977335 26425 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:00.981699 26425 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } }
I20250809 02:05:00.982389 26425 leader_election.cc:304] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f155abab92e41fe90c86d52b9b170ba; no voters:
I20250809 02:05:00.984683 26425 leader_election.cc:290] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:05:00.985488 26430 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:05:00.988274 26430 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 1 LEADER]: Becoming Leader. State: Replica: 2f155abab92e41fe90c86d52b9b170ba, State: Running, Role: LEADER
I20250809 02:05:00.989063 26430 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } }
I20250809 02:05:00.990558 26425 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:05:01.000365 26432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f155abab92e41fe90c86d52b9b170ba. Latest consensus state: current_term: 1 leader_uuid: "2f155abab92e41fe90c86d52b9b170ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } } }
I20250809 02:05:01.000340 26431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f155abab92e41fe90c86d52b9b170ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } } }
I20250809 02:05:01.001323 26431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [sys.catalog]: This master's current role is: LEADER
I20250809 02:05:01.001323 26432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [sys.catalog]: This master's current role is: LEADER
I20250809 02:05:01.009939 26438 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:05:01.021768 26438 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:05:01.038550 26438 catalog_manager.cc:1349] Generated new cluster ID: 9e8c8d0d68dc46819edde06004e918df
I20250809 02:05:01.038913 26438 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:05:01.053651 26438 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 02:05:01.056028 26438 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:05:01.069478 26438 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba: Generated new TSK 0
I20250809 02:05:01.070479 26438 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 02:05:01.080881 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:0
--local_ip_for_outbound_sockets=127.25.185.193
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:44687
--builtin_ntp_servers=127.25.185.212:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250809 02:05:01.401304 26449 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:01.401803 26449 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:01.402292 26449 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:01.434458 26449 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:01.435353 26449 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:05:01.472270 26449 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:44687
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:01.473657 26449 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:01.475359 26449 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:01.488689 26455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:01.491504 26456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:02.892161 26454 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 26449
W20250809 02:05:02.931563 26449 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.442s user 0.482s sys 0.957s
W20250809 02:05:02.933121 26449 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.444s user 0.483s sys 0.958s
W20250809 02:05:02.934557 26458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:02.937057 26457 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250809 02:05:02.937064 26449 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:02.938283 26449 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:02.940469 26449 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:02.941845 26449 hybrid_clock.cc:648] HybridClock initialized: now 1754705102941800 us; error 54 us; skew 500 ppm
I20250809 02:05:02.942641 26449 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:02.949399 26449 webserver.cc:489] Webserver started at http://127.25.185.193:35057/ using document root <none> and password file <none>
I20250809 02:05:02.950332 26449 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:02.950538 26449 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:02.950987 26449 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:02.955365 26449 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "96d09e651b9d450b97762273a1c7fc77"
format_stamp: "Formatted at 2025-08-09 02:05:02 on dist-test-slave-24ff"
I20250809 02:05:02.956523 26449 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "96d09e651b9d450b97762273a1c7fc77"
format_stamp: "Formatted at 2025-08-09 02:05:02 on dist-test-slave-24ff"
I20250809 02:05:02.964542 26449 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250809 02:05:02.971212 26465 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:02.972373 26449 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250809 02:05:02.972690 26449 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "96d09e651b9d450b97762273a1c7fc77"
format_stamp: "Formatted at 2025-08-09 02:05:02 on dist-test-slave-24ff"
I20250809 02:05:02.973030 26449 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:03.023221 26449 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:03.024631 26449 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:03.025068 26449 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:03.028142 26449 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:03.032790 26449 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:03.033002 26449 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:03.033257 26449 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:03.033449 26449 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:03.220839 26449 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:42963
I20250809 02:05:03.220949 26577 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:42963 every 8 connection(s)
I20250809 02:05:03.223542 26449 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 02:05:03.232776 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 26449
I20250809 02:05:03.233297 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 02:05:03.241070 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.194:0
--local_ip_for_outbound_sockets=127.25.185.194
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:44687
--builtin_ntp_servers=127.25.185.212:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:05:03.258358 26578 heartbeater.cc:344] Connected to a master server at 127.25.185.254:44687
I20250809 02:05:03.258917 26578 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:03.260288 26578 heartbeater.cc:507] Master 127.25.185.254:44687 requested a full tablet report, sending...
I20250809 02:05:03.263561 26390 ts_manager.cc:194] Registered new tserver with Master: 96d09e651b9d450b97762273a1c7fc77 (127.25.185.193:42963)
I20250809 02:05:03.266474 26390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:44495
W20250809 02:05:03.557713 26582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:03.558271 26582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:03.558874 26582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:03.589514 26582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:03.590379 26582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.194
I20250809 02:05:03.624805 26582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:44687
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.194
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:03.626192 26582 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:03.627719 26582 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:03.641274 26588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:04.270622 26578 heartbeater.cc:499] Master 127.25.185.254:44687 was elected leader, sending a full tablet report...
W20250809 02:05:03.641716 26589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:04.854873 26582 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.214s user 0.303s sys 0.911s
W20250809 02:05:04.855396 26582 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.215s user 0.303s sys 0.911s
W20250809 02:05:04.855662 26590 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1209 milliseconds
W20250809 02:05:04.856644 26591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:04.856707 26582 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:04.857966 26582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:04.860729 26582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:04.862092 26582 hybrid_clock.cc:648] HybridClock initialized: now 1754705104862060 us; error 41 us; skew 500 ppm
I20250809 02:05:04.862923 26582 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:04.870828 26582 webserver.cc:489] Webserver started at http://127.25.185.194:34887/ using document root <none> and password file <none>
I20250809 02:05:04.871739 26582 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:04.871960 26582 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:04.872399 26582 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:04.877241 26582 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f948ab3509494e3b983291c60a264978"
format_stamp: "Formatted at 2025-08-09 02:05:04 on dist-test-slave-24ff"
I20250809 02:05:04.878320 26582 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f948ab3509494e3b983291c60a264978"
format_stamp: "Formatted at 2025-08-09 02:05:04 on dist-test-slave-24ff"
I20250809 02:05:04.886372 26582 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250809 02:05:04.892145 26598 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:04.893221 26582 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250809 02:05:04.893514 26582 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f948ab3509494e3b983291c60a264978"
format_stamp: "Formatted at 2025-08-09 02:05:04 on dist-test-slave-24ff"
I20250809 02:05:04.893883 26582 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:04.982445 26582 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:04.983968 26582 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:04.984393 26582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:04.986948 26582 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:04.991364 26582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:04.991566 26582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:04.991792 26582 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:04.991990 26582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:05.139106 26582 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.194:44771
I20250809 02:05:05.139217 26710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.194:44771 every 8 connection(s)
I20250809 02:05:05.141605 26582 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 02:05:05.150084 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 26582
I20250809 02:05:05.150655 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 02:05:05.157478 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.195:0
--local_ip_for_outbound_sockets=127.25.185.195
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:44687
--builtin_ntp_servers=127.25.185.212:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:05:05.166445 26711 heartbeater.cc:344] Connected to a master server at 127.25.185.254:44687
I20250809 02:05:05.166882 26711 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:05.167902 26711 heartbeater.cc:507] Master 127.25.185.254:44687 requested a full tablet report, sending...
I20250809 02:05:05.170140 26390 ts_manager.cc:194] Registered new tserver with Master: f948ab3509494e3b983291c60a264978 (127.25.185.194:44771)
I20250809 02:05:05.171972 26390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.194:52405
W20250809 02:05:05.462266 26715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:05.462777 26715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:05.463258 26715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:05.494415 26715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:05.495266 26715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.195
I20250809 02:05:05.529979 26715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:44687
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.195
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:05.531257 26715 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:05.532701 26715 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:05.545456 26721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:06.175858 26711 heartbeater.cc:499] Master 127.25.185.254:44687 was elected leader, sending a full tablet report...
W20250809 02:05:05.545471 26722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:06.890571 26724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:06.894282 26723 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1348 milliseconds
W20250809 02:05:06.894733 26715 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.349s user 0.004s sys 0.004s
W20250809 02:05:06.895015 26715 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.350s user 0.004s sys 0.005s
I20250809 02:05:06.895210 26715 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:06.896190 26715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:06.898748 26715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:06.900214 26715 hybrid_clock.cc:648] HybridClock initialized: now 1754705106900183 us; error 50 us; skew 500 ppm
I20250809 02:05:06.901036 26715 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:06.913375 26715 webserver.cc:489] Webserver started at http://127.25.185.195:44687/ using document root <none> and password file <none>
I20250809 02:05:06.914343 26715 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:06.914563 26715 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:06.915026 26715 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:06.919608 26715 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "75b03544cc9b4dc1b59832de7b400910"
format_stamp: "Formatted at 2025-08-09 02:05:06 on dist-test-slave-24ff"
I20250809 02:05:06.920746 26715 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "75b03544cc9b4dc1b59832de7b400910"
format_stamp: "Formatted at 2025-08-09 02:05:06 on dist-test-slave-24ff"
W20250809 02:05:06.922230 26421 messenger.cc:191] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.25.185.254:44687
I20250809 02:05:06.928782 26715 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250809 02:05:06.935062 26731 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:06.936180 26715 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250809 02:05:06.936511 26715 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "75b03544cc9b4dc1b59832de7b400910"
format_stamp: "Formatted at 2025-08-09 02:05:06 on dist-test-slave-24ff"
I20250809 02:05:06.936823 26715 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:07.020203 26715 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:07.021718 26715 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:07.022148 26715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:07.025067 26715 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:07.029032 26715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:07.029239 26715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:07.029479 26715 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:07.029637 26715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:07.187214 26715 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.195:35225
I20250809 02:05:07.187280 26843 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.195:35225 every 8 connection(s)
I20250809 02:05:07.189648 26715 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 02:05:07.194970 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 26715
I20250809 02:05:07.195488 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 02:05:07.201790 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.196:0
--local_ip_for_outbound_sockets=127.25.185.196
--webserver_interface=127.25.185.196
--webserver_port=0
--tserver_master_addrs=127.25.185.254:44687
--builtin_ntp_servers=127.25.185.212:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:05:07.212507 26844 heartbeater.cc:344] Connected to a master server at 127.25.185.254:44687
I20250809 02:05:07.213049 26844 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:07.214916 26844 heartbeater.cc:507] Master 127.25.185.254:44687 requested a full tablet report, sending...
I20250809 02:05:07.217314 26390 ts_manager.cc:194] Registered new tserver with Master: 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195:35225)
I20250809 02:05:07.218922 26390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.195:53565
W20250809 02:05:07.522130 26848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:07.522650 26848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:07.523211 26848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:07.555370 26848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:07.556216 26848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.196
I20250809 02:05:07.591087 26848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.185.196
--webserver_port=0
--tserver_master_addrs=127.25.185.254:44687
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.196
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:07.592607 26848 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:07.594161 26848 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:07.608438 26854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:08.222090 26844 heartbeater.cc:499] Master 127.25.185.254:44687 was elected leader, sending a full tablet report...
W20250809 02:05:07.610085 26855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:08.826283 26856 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250809 02:05:08.828576 26857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:08.831257 26848 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.223s user 0.386s sys 0.837s
W20250809 02:05:08.831496 26848 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.223s user 0.386s sys 0.837s
I20250809 02:05:08.831662 26848 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:08.832616 26848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:08.848982 26848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:08.850404 26848 hybrid_clock.cc:648] HybridClock initialized: now 1754705108850368 us; error 58 us; skew 500 ppm
I20250809 02:05:08.851274 26848 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:08.859390 26848 webserver.cc:489] Webserver started at http://127.25.185.196:38563/ using document root <none> and password file <none>
I20250809 02:05:08.860383 26848 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:08.860610 26848 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:08.861047 26848 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:08.865500 26848 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8ce76fbab4c74ad3827a525044801d85"
format_stamp: "Formatted at 2025-08-09 02:05:08 on dist-test-slave-24ff"
I20250809 02:05:08.866652 26848 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8ce76fbab4c74ad3827a525044801d85"
format_stamp: "Formatted at 2025-08-09 02:05:08 on dist-test-slave-24ff"
I20250809 02:05:08.875376 26848 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.002s
I20250809 02:05:08.881996 26865 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:08.883208 26848 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250809 02:05:08.883540 26848 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8ce76fbab4c74ad3827a525044801d85"
format_stamp: "Formatted at 2025-08-09 02:05:08 on dist-test-slave-24ff"
I20250809 02:05:08.883869 26848 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:08.960479 26848 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:08.961920 26848 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:08.962332 26848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:08.964923 26848 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:08.969626 26848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:08.969825 26848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:08.970083 26848 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:08.970227 26848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:09.120066 26848 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.196:44113
I20250809 02:05:09.120718 26977 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.196:44113 every 8 connection(s)
I20250809 02:05:09.122838 26848 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250809 02:05:09.133152 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 26848
I20250809 02:05:09.133682 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250809 02:05:09.147765 26978 heartbeater.cc:344] Connected to a master server at 127.25.185.254:44687
I20250809 02:05:09.148577 26978 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:09.149817 26978 heartbeater.cc:507] Master 127.25.185.254:44687 requested a full tablet report, sending...
I20250809 02:05:09.151961 26390 ts_manager.cc:194] Registered new tserver with Master: 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196:44113)
I20250809 02:05:09.153364 26390 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.196:41683
I20250809 02:05:09.155819 26343 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250809 02:05:09.199366 26390 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37840:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250809 02:05:09.282874 26779 tablet_service.cc:1468] Processing CreateTablet for tablet 14e76c181acb4439b9c57e6a94bc98d4 (DEFAULT_TABLE table=TestTable [id=5b09bc555fa344bba667d9fdbbcea120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:05:09.285269 26779 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e76c181acb4439b9c57e6a94bc98d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:09.286098 26913 tablet_service.cc:1468] Processing CreateTablet for tablet 14e76c181acb4439b9c57e6a94bc98d4 (DEFAULT_TABLE table=TestTable [id=5b09bc555fa344bba667d9fdbbcea120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:05:09.287683 26913 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e76c181acb4439b9c57e6a94bc98d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:09.287343 26646 tablet_service.cc:1468] Processing CreateTablet for tablet 14e76c181acb4439b9c57e6a94bc98d4 (DEFAULT_TABLE table=TestTable [id=5b09bc555fa344bba667d9fdbbcea120]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:05:09.289214 26646 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e76c181acb4439b9c57e6a94bc98d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:09.315852 26997 tablet_bootstrap.cc:492] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85: Bootstrap starting.
I20250809 02:05:09.315902 26998 tablet_bootstrap.cc:492] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910: Bootstrap starting.
I20250809 02:05:09.319428 26999 tablet_bootstrap.cc:492] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978: Bootstrap starting.
I20250809 02:05:09.324062 26998 tablet_bootstrap.cc:654] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:09.324160 26997 tablet_bootstrap.cc:654] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:09.325984 26999 tablet_bootstrap.cc:654] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:09.326851 26998 log.cc:826] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:09.328044 26997 log.cc:826] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:09.328397 26999 log.cc:826] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:09.333174 26998 tablet_bootstrap.cc:492] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910: No bootstrap required, opened a new log
I20250809 02:05:09.333667 26998 ts_tablet_manager.cc:1397] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20250809 02:05:09.335142 26999 tablet_bootstrap.cc:492] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978: No bootstrap required, opened a new log
I20250809 02:05:09.335141 26997 tablet_bootstrap.cc:492] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85: No bootstrap required, opened a new log
I20250809 02:05:09.335878 26999 ts_tablet_manager.cc:1397] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.010s
I20250809 02:05:09.336026 26997 ts_tablet_manager.cc:1397] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.009s
I20250809 02:05:09.355167 26997 raft_consensus.cc:357] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:09.356452 26997 raft_consensus.cc:383] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:09.356726 26997 raft_consensus.cc:738] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ce76fbab4c74ad3827a525044801d85, State: Initialized, Role: FOLLOWER
I20250809 02:05:09.357470 26997 consensus_queue.cc:260] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:09.361611 26978 heartbeater.cc:499] Master 127.25.185.254:44687 was elected leader, sending a full tablet report...
I20250809 02:05:09.362205 26998 raft_consensus.cc:357] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:09.362865 26997 ts_tablet_manager.cc:1428] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85: Time spent starting tablet: real 0.027s user 0.027s sys 0.001s
I20250809 02:05:09.363188 26998 raft_consensus.cc:383] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:09.362241 26999 raft_consensus.cc:357] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:09.363538 26999 raft_consensus.cc:383] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:09.363556 26998 raft_consensus.cc:738] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75b03544cc9b4dc1b59832de7b400910, State: Initialized, Role: FOLLOWER
I20250809 02:05:09.363840 26999 raft_consensus.cc:738] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f948ab3509494e3b983291c60a264978, State: Initialized, Role: FOLLOWER
I20250809 02:05:09.364730 26999 consensus_queue.cc:260] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:09.364378 26998 consensus_queue.cc:260] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:09.369736 26998 ts_tablet_manager.cc:1428] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910: Time spent starting tablet: real 0.036s user 0.024s sys 0.010s
I20250809 02:05:09.374132 26999 ts_tablet_manager.cc:1428] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978: Time spent starting tablet: real 0.038s user 0.031s sys 0.006s
W20250809 02:05:09.377899 26979 tablet.cc:2378] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 02:05:09.401106 26712 tablet.cc:2378] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 02:05:09.447140 26845 tablet.cc:2378] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:05:09.511118 27003 raft_consensus.cc:491] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:05:09.511628 27003 raft_consensus.cc:513] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:09.513926 27003 leader_election.cc:290] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195:35225), f948ab3509494e3b983291c60a264978 (127.25.185.194:44771)
I20250809 02:05:09.523087 26799 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e76c181acb4439b9c57e6a94bc98d4" candidate_uuid: "8ce76fbab4c74ad3827a525044801d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b03544cc9b4dc1b59832de7b400910" is_pre_election: true
I20250809 02:05:09.523877 26799 raft_consensus.cc:2466] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce76fbab4c74ad3827a525044801d85 in term 0.
I20250809 02:05:09.525059 26866 leader_election.cc:304] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75b03544cc9b4dc1b59832de7b400910, 8ce76fbab4c74ad3827a525044801d85; no voters:
I20250809 02:05:09.525290 26666 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e76c181acb4439b9c57e6a94bc98d4" candidate_uuid: "8ce76fbab4c74ad3827a525044801d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f948ab3509494e3b983291c60a264978" is_pre_election: true
I20250809 02:05:09.525779 27003 raft_consensus.cc:2802] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 02:05:09.526021 27003 raft_consensus.cc:491] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 02:05:09.525959 26666 raft_consensus.cc:2466] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ce76fbab4c74ad3827a525044801d85 in term 0.
I20250809 02:05:09.526273 27003 raft_consensus.cc:3058] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:09.530942 27003 raft_consensus.cc:513] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:09.532512 27003 leader_election.cc:290] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [CANDIDATE]: Term 1 election: Requested vote from peers 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195:35225), f948ab3509494e3b983291c60a264978 (127.25.185.194:44771)
I20250809 02:05:09.533252 26799 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e76c181acb4439b9c57e6a94bc98d4" candidate_uuid: "8ce76fbab4c74ad3827a525044801d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75b03544cc9b4dc1b59832de7b400910"
I20250809 02:05:09.533476 26666 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e76c181acb4439b9c57e6a94bc98d4" candidate_uuid: "8ce76fbab4c74ad3827a525044801d85" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f948ab3509494e3b983291c60a264978"
I20250809 02:05:09.533679 26799 raft_consensus.cc:3058] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:09.533898 26666 raft_consensus.cc:3058] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:09.538161 26799 raft_consensus.cc:2466] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce76fbab4c74ad3827a525044801d85 in term 1.
I20250809 02:05:09.538240 26666 raft_consensus.cc:2466] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ce76fbab4c74ad3827a525044801d85 in term 1.
I20250809 02:05:09.539058 26866 leader_election.cc:304] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75b03544cc9b4dc1b59832de7b400910, 8ce76fbab4c74ad3827a525044801d85; no voters:
I20250809 02:05:09.539780 27003 raft_consensus.cc:2802] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:05:09.541239 27003 raft_consensus.cc:695] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [term 1 LEADER]: Becoming Leader. State: Replica: 8ce76fbab4c74ad3827a525044801d85, State: Running, Role: LEADER
I20250809 02:05:09.541931 27003 consensus_queue.cc:237] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:09.552460 26388 catalog_manager.cc:5582] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196). New cstate: current_term: 1 leader_uuid: "8ce76fbab4c74ad3827a525044801d85" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } health_report { overall_health: HEALTHY } } }
I20250809 02:05:09.576812 26343 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250809 02:05:09.580644 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f948ab3509494e3b983291c60a264978 to finish bootstrapping
I20250809 02:05:09.594718 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 75b03544cc9b4dc1b59832de7b400910 to finish bootstrapping
I20250809 02:05:09.604897 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8ce76fbab4c74ad3827a525044801d85 to finish bootstrapping
I20250809 02:05:09.615507 26343 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250809 02:05:09.618767 26343 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "14e76c181acb4439b9c57e6a94bc98d4"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: LEADER
}
}
ts_infos {
permanent_uuid: "75b03544cc9b4dc1b59832de7b400910"
rpc_addresses {
host: "127.25.185.195"
port: 35225
}
}
ts_infos {
permanent_uuid: "f948ab3509494e3b983291c60a264978"
rpc_addresses {
host: "127.25.185.194"
port: 44771
}
}
ts_infos {
permanent_uuid: "8ce76fbab4c74ad3827a525044801d85"
rpc_addresses {
host: "127.25.185.196"
port: 44113
}
}
I20250809 02:05:09.954909 27003 consensus_queue.cc:1035] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 02:05:09.976744 27016 consensus_queue.cc:1035] T 14e76c181acb4439b9c57e6a94bc98d4 P 8ce76fbab4c74ad3827a525044801d85 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 02:05:09.978981 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 26848
W20250809 02:05:10.008385 26732 connection.cc:537] server connection from 127.25.185.196:40575 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250809 02:05:10.008751 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 26357
I20250809 02:05:10.037488 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:44687
--webserver_interface=127.25.185.254
--webserver_port=38791
--builtin_ntp_servers=127.25.185.212:45293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:44687 with env {}
W20250809 02:05:10.303196 26578 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:44687 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:44687: connect: Connection refused (error 111)
W20250809 02:05:10.337412 27020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:10.338052 27020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:10.338506 27020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:10.368796 27020 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:05:10.369120 27020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:10.369372 27020 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:05:10.369607 27020 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:05:10.404172 27020 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:45293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:44687
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:44687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=38791
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:10.405525 27020 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:10.407167 27020 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:10.417141 27027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:11.000964 26711 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:44687 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:44687: connect: Connection refused (error 111)
W20250809 02:05:11.004745 26844 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:44687 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:44687: connect: Connection refused (error 111)
I20250809 02:05:11.473946 27037 raft_consensus.cc:491] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8ce76fbab4c74ad3827a525044801d85)
I20250809 02:05:11.474735 27037 raft_consensus.cc:513] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:11.491607 27039 raft_consensus.cc:491] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 8ce76fbab4c74ad3827a525044801d85)
I20250809 02:05:11.496970 27037 leader_election.cc:290] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195:35225), 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196:44113)
I20250809 02:05:11.502139 27039 raft_consensus.cc:513] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:11.512226 27039 leader_election.cc:290] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f948ab3509494e3b983291c60a264978 (127.25.185.194:44771), 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196:44113)
W20250809 02:05:11.513337 26600 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.185.196:44113: connect: Connection refused (error 111)
W20250809 02:05:11.535312 26733 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.185.196:44113: connect: Connection refused (error 111)
W20250809 02:05:11.537659 26600 leader_election.cc:336] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196:44113): Network error: Client connection negotiation failed: client connection to 127.25.185.196:44113: connect: Connection refused (error 111)
I20250809 02:05:11.545657 26666 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e76c181acb4439b9c57e6a94bc98d4" candidate_uuid: "75b03544cc9b4dc1b59832de7b400910" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f948ab3509494e3b983291c60a264978" is_pre_election: true
I20250809 02:05:11.546277 26666 raft_consensus.cc:2466] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75b03544cc9b4dc1b59832de7b400910 in term 1.
I20250809 02:05:11.547561 26733 leader_election.cc:304] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75b03544cc9b4dc1b59832de7b400910, f948ab3509494e3b983291c60a264978; no voters:
I20250809 02:05:11.548395 27039 raft_consensus.cc:2802] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250809 02:05:11.548786 27039 raft_consensus.cc:491] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 8ce76fbab4c74ad3827a525044801d85)
I20250809 02:05:11.549273 27039 raft_consensus.cc:3058] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:05:11.552870 26799 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e76c181acb4439b9c57e6a94bc98d4" candidate_uuid: "f948ab3509494e3b983291c60a264978" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "75b03544cc9b4dc1b59832de7b400910" is_pre_election: true
W20250809 02:05:11.553709 26733 leader_election.cc:336] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196:44113): Network error: Client connection negotiation failed: client connection to 127.25.185.196:44113: connect: Connection refused (error 111)
I20250809 02:05:11.557559 27039 raft_consensus.cc:513] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:11.559316 26799 raft_consensus.cc:2391] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f948ab3509494e3b983291c60a264978 in current term 2: Already voted for candidate 75b03544cc9b4dc1b59832de7b400910 in this term.
I20250809 02:05:11.564714 26599 leader_election.cc:304] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f948ab3509494e3b983291c60a264978; no voters: 75b03544cc9b4dc1b59832de7b400910, 8ce76fbab4c74ad3827a525044801d85
I20250809 02:05:11.566609 27037 raft_consensus.cc:3058] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:05:11.576792 27037 raft_consensus.cc:2747] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250809 02:05:11.588305 26733 leader_election.cc:336] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196:44113): Network error: Client connection negotiation failed: client connection to 127.25.185.196:44113: connect: Connection refused (error 111)
I20250809 02:05:11.580380 26666 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e76c181acb4439b9c57e6a94bc98d4" candidate_uuid: "75b03544cc9b4dc1b59832de7b400910" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f948ab3509494e3b983291c60a264978"
I20250809 02:05:11.598909 27039 leader_election.cc:290] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [CANDIDATE]: Term 2 election: Requested vote from peers f948ab3509494e3b983291c60a264978 (127.25.185.194:44771), 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196:44113)
I20250809 02:05:11.599912 26666 raft_consensus.cc:2466] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75b03544cc9b4dc1b59832de7b400910 in term 2.
I20250809 02:05:11.602360 26733 leader_election.cc:304] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75b03544cc9b4dc1b59832de7b400910, f948ab3509494e3b983291c60a264978; no voters: 8ce76fbab4c74ad3827a525044801d85
I20250809 02:05:11.603945 27039 raft_consensus.cc:2802] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 02:05:11.608845 27039 raft_consensus.cc:695] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 2 LEADER]: Becoming Leader. State: Replica: 75b03544cc9b4dc1b59832de7b400910, State: Running, Role: LEADER
I20250809 02:05:11.611227 27039 consensus_queue.cc:237] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
W20250809 02:05:11.821158 27026 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 27020
W20250809 02:05:11.840323 27026 kernel_stack_watchdog.cc:198] Thread 27020 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 02:05:11.840929 27020 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.424s user 0.525s sys 0.888s
W20250809 02:05:11.841480 27020 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.424s user 0.526s sys 0.888s
W20250809 02:05:10.417439 27028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:11.842833 27029 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1425 milliseconds
I20250809 02:05:11.843514 27020 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250809 02:05:11.843591 27030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:11.847002 27020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:11.849538 27020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:11.850883 27020 hybrid_clock.cc:648] HybridClock initialized: now 1754705111850845 us; error 47 us; skew 500 ppm
I20250809 02:05:11.851675 27020 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:11.857599 27020 webserver.cc:489] Webserver started at http://127.25.185.254:38791/ using document root <none> and password file <none>
I20250809 02:05:11.858530 27020 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:11.858784 27020 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:11.866437 27020 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20250809 02:05:11.871086 27052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:11.872165 27020 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250809 02:05:11.872491 27020 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2f155abab92e41fe90c86d52b9b170ba"
format_stamp: "Formatted at 2025-08-09 02:05:00 on dist-test-slave-24ff"
I20250809 02:05:11.874406 27020 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:11.927234 27020 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:11.928694 27020 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:11.929131 27020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:11.997924 26666 raft_consensus.cc:1273] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 2 FOLLOWER]: Refusing update from remote peer 75b03544cc9b4dc1b59832de7b400910: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250809 02:05:11.999632 27039 consensus_queue.cc:1035] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250809 02:05:12.004992 27020 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:44687
I20250809 02:05:12.005074 27104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:44687 every 8 connection(s)
I20250809 02:05:12.015965 27020 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
W20250809 02:05:12.016309 26733 consensus_peers.cc:489] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 -> Peer 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196:44113): Couldn't send request to peer 8ce76fbab4c74ad3827a525044801d85. Status: Network error: Client connection negotiation failed: client connection to 127.25.185.196:44113: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250809 02:05:12.018568 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 27020
I20250809 02:05:12.019255 26343 kudu-admin-test.cc:735] Forcing unsafe config change on tserver f948ab3509494e3b983291c60a264978
I20250809 02:05:12.032486 27110 sys_catalog.cc:263] Verifying existing consensus state
I20250809 02:05:12.040910 27110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba: Bootstrap starting.
I20250809 02:05:12.042711 26844 heartbeater.cc:344] Connected to a master server at 127.25.185.254:44687
I20250809 02:05:12.044520 26711 heartbeater.cc:344] Connected to a master server at 127.25.185.254:44687
I20250809 02:05:12.081247 27110 log.cc:826] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:12.103363 27110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:05:12.104146 27110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba: Bootstrap complete.
I20250809 02:05:12.122272 27110 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } }
I20250809 02:05:12.124408 27110 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f155abab92e41fe90c86d52b9b170ba, State: Initialized, Role: FOLLOWER
I20250809 02:05:12.125190 27110 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } }
I20250809 02:05:12.125694 27110 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:05:12.125990 27110 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:05:12.126318 27110 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:05:12.131968 27110 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } }
I20250809 02:05:12.132659 27110 leader_election.cc:304] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f155abab92e41fe90c86d52b9b170ba; no voters:
I20250809 02:05:12.134863 27110 leader_election.cc:290] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [CANDIDATE]: Term 2 election: Requested vote from peers
I20250809 02:05:12.135133 27114 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 2 FOLLOWER]: Leader election won for term 2
I20250809 02:05:12.143734 27114 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [term 2 LEADER]: Becoming Leader. State: Replica: 2f155abab92e41fe90c86d52b9b170ba, State: Running, Role: LEADER
I20250809 02:05:12.145041 27114 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } }
I20250809 02:05:12.149829 27110 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:05:12.162714 27115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f155abab92e41fe90c86d52b9b170ba. Latest consensus state: current_term: 2 leader_uuid: "2f155abab92e41fe90c86d52b9b170ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } } }
I20250809 02:05:12.163527 27115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [sys.catalog]: This master's current role is: LEADER
I20250809 02:05:12.162745 27116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2f155abab92e41fe90c86d52b9b170ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f155abab92e41fe90c86d52b9b170ba" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 44687 } } }
I20250809 02:05:12.164737 27116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba [sys.catalog]: This master's current role is: LEADER
I20250809 02:05:12.173103 27122 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:05:12.197108 27122 catalog_manager.cc:671] Loaded metadata for table TestTable [id=5b09bc555fa344bba667d9fdbbcea120]
I20250809 02:05:12.206720 27122 tablet_loader.cc:96] loaded metadata for tablet 14e76c181acb4439b9c57e6a94bc98d4 (table TestTable [id=5b09bc555fa344bba667d9fdbbcea120])
I20250809 02:05:12.209246 27122 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:05:12.215488 27122 catalog_manager.cc:1261] Loaded cluster ID: 9e8c8d0d68dc46819edde06004e918df
I20250809 02:05:12.215965 27122 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:05:12.226167 27122 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:05:12.232437 27122 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2f155abab92e41fe90c86d52b9b170ba: Loaded TSK: 0
I20250809 02:05:12.234362 27122 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 02:05:12.338150 26578 heartbeater.cc:344] Connected to a master server at 127.25.185.254:44687
I20250809 02:05:12.343945 27069 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" instance_seqno: 1754705103177804) as {username='slave'} at 127.25.185.193:54695; Asking this server to re-register.
I20250809 02:05:12.345325 26578 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:12.346035 26578 heartbeater.cc:507] Master 127.25.185.254:44687 requested a full tablet report, sending...
I20250809 02:05:12.348526 27069 ts_manager.cc:194] Registered new tserver with Master: 96d09e651b9d450b97762273a1c7fc77 (127.25.185.193:42963)
W20250809 02:05:12.412720 27112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:12.413317 27112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:12.444698 27112 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250809 02:05:13.028100 27100 messenger.cc:191] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.25.185.254:44687
I20250809 02:05:13.052440 27069 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" instance_seqno: 1754705107149955) as {username='slave'} at 127.25.185.195:44077; Asking this server to re-register.
I20250809 02:05:13.056509 26844 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:13.057367 26844 heartbeater.cc:507] Master 127.25.185.254:44687 requested a full tablet report, sending...
I20250809 02:05:13.057672 27069 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f948ab3509494e3b983291c60a264978" instance_seqno: 1754705105103846) as {username='slave'} at 127.25.185.194:35847; Asking this server to re-register.
I20250809 02:05:13.059316 26711 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:13.059934 26711 heartbeater.cc:507] Master 127.25.185.254:44687 requested a full tablet report, sending...
I20250809 02:05:13.063611 27068 ts_manager.cc:194] Registered new tserver with Master: 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195:35225)
I20250809 02:05:13.068240 27069 ts_manager.cc:194] Registered new tserver with Master: f948ab3509494e3b983291c60a264978 (127.25.185.194:44771)
I20250809 02:05:13.069690 27068 catalog_manager.cc:5582] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 reported cstate change: term changed from 1 to 2, leader changed from 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196) to 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195). New cstate: current_term: 2 leader_uuid: "75b03544cc9b4dc1b59832de7b400910" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } health_report { overall_health: UNKNOWN } } }
W20250809 02:05:13.889854 27138 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 27112
W20250809 02:05:13.890375 27138 kernel_stack_watchdog.cc:198] Thread 27112 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 02:05:14.093417 27112 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.606s user 0.513s sys 1.060s
W20250809 02:05:14.234350 27112 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.748s user 0.513s sys 1.071s
I20250809 02:05:14.286939 26666 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "f948ab3509494e3b983291c60a264978"
tablet_id: "14e76c181acb4439b9c57e6a94bc98d4"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "75b03544cc9b4dc1b59832de7b400910"
}
peers {
permanent_uuid: "f948ab3509494e3b983291c60a264978"
}
}
from {username='slave'} at 127.0.0.1:34752
W20250809 02:05:14.288414 26666 raft_consensus.cc:2216] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } unsafe_config_change: true
I20250809 02:05:14.289359 26666 raft_consensus.cc:3058] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 2 FOLLOWER]: Advancing to term 3
I20250809 02:05:14.642408 26666 raft_consensus.cc:1238] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 3 FOLLOWER]: Rejecting Update request from peer 75b03544cc9b4dc1b59832de7b400910 for earlier term 2. Current term is 3. Ops: []
I20250809 02:05:14.643579 27039 consensus_queue.cc:1046] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250809 02:05:14.644647 27039 raft_consensus.cc:3053] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 2 LEADER]: Stepping down as leader of term 2
I20250809 02:05:14.644865 27039 raft_consensus.cc:738] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 75b03544cc9b4dc1b59832de7b400910, State: Running, Role: LEADER
I20250809 02:05:14.645476 27039 consensus_queue.cc:260] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } }
I20250809 02:05:14.646358 27039 raft_consensus.cc:3058] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 2 FOLLOWER]: Advancing to term 3
W20250809 02:05:15.091975 27138 debug-util.cc:398] Leaking SignalData structure 0x7b08000379e0 after lost signal to thread 27112
I20250809 02:05:15.796830 27167 raft_consensus.cc:491] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250809 02:05:15.797259 27167 raft_consensus.cc:513] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } unsafe_config_change: true
I20250809 02:05:15.798367 27167 leader_election.cc:290] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195:35225)
I20250809 02:05:15.799827 26799 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e76c181acb4439b9c57e6a94bc98d4" candidate_uuid: "f948ab3509494e3b983291c60a264978" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "75b03544cc9b4dc1b59832de7b400910" is_pre_election: true
I20250809 02:05:15.800586 26799 raft_consensus.cc:2466] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f948ab3509494e3b983291c60a264978 in term 3.
I20250809 02:05:15.801779 26599 leader_election.cc:304] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 75b03544cc9b4dc1b59832de7b400910, f948ab3509494e3b983291c60a264978; no voters:
I20250809 02:05:15.802405 27167 raft_consensus.cc:2802] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250809 02:05:15.802701 27167 raft_consensus.cc:491] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250809 02:05:15.802934 27167 raft_consensus.cc:3058] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 3 FOLLOWER]: Advancing to term 4
I20250809 02:05:15.807169 27167 raft_consensus.cc:513] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } unsafe_config_change: true
I20250809 02:05:15.808171 27167 leader_election.cc:290] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [CANDIDATE]: Term 4 election: Requested vote from peers 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195:35225)
I20250809 02:05:15.809181 26799 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14e76c181acb4439b9c57e6a94bc98d4" candidate_uuid: "f948ab3509494e3b983291c60a264978" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "75b03544cc9b4dc1b59832de7b400910"
I20250809 02:05:15.809715 26799 raft_consensus.cc:3058] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 3 FOLLOWER]: Advancing to term 4
I20250809 02:05:15.814356 26799 raft_consensus.cc:2466] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f948ab3509494e3b983291c60a264978 in term 4.
I20250809 02:05:15.815205 26599 leader_election.cc:304] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 75b03544cc9b4dc1b59832de7b400910, f948ab3509494e3b983291c60a264978; no voters:
I20250809 02:05:15.815798 27167 raft_consensus.cc:2802] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 4 FOLLOWER]: Leader election won for term 4
I20250809 02:05:15.816615 27167 raft_consensus.cc:695] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 4 LEADER]: Becoming Leader. State: Replica: f948ab3509494e3b983291c60a264978, State: Running, Role: LEADER
I20250809 02:05:15.817268 27167 consensus_queue.cc:237] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } unsafe_config_change: true
I20250809 02:05:15.823465 27068 catalog_manager.cc:5582] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 reported cstate change: term changed from 2 to 4, leader changed from 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195) to f948ab3509494e3b983291c60a264978 (127.25.185.194), now has a pending config: VOTER 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195), VOTER f948ab3509494e3b983291c60a264978 (127.25.185.194). New cstate: current_term: 4 leader_uuid: "f948ab3509494e3b983291c60a264978" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ce76fbab4c74ad3827a525044801d85" member_type: VOTER last_known_addr { host: "127.25.185.196" port: 44113 } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } unsafe_config_change: true }
I20250809 02:05:16.286177 26799 raft_consensus.cc:1273] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 4 FOLLOWER]: Refusing update from remote peer f948ab3509494e3b983291c60a264978: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250809 02:05:16.287446 27167 consensus_queue.cc:1035] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250809 02:05:16.295840 27168 raft_consensus.cc:2953] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } unsafe_config_change: true }
I20250809 02:05:16.296921 26799 raft_consensus.cc:2953] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } unsafe_config_change: true }
I20250809 02:05:16.307013 27068 catalog_manager.cc:5582] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 reported cstate change: config changed from index -1 to 3, VOTER 8ce76fbab4c74ad3827a525044801d85 (127.25.185.196) evicted, no longer has a pending config: VOTER 75b03544cc9b4dc1b59832de7b400910 (127.25.185.195), VOTER f948ab3509494e3b983291c60a264978 (127.25.185.194). New cstate: current_term: 4 leader_uuid: "f948ab3509494e3b983291c60a264978" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } unsafe_config_change: true }
W20250809 02:05:16.313792 27068 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 14e76c181acb4439b9c57e6a94bc98d4 on TS 8ce76fbab4c74ad3827a525044801d85: Not found: failed to reset TS proxy: Could not find TS for UUID 8ce76fbab4c74ad3827a525044801d85
I20250809 02:05:16.329461 26666 consensus_queue.cc:237] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: true } } unsafe_config_change: true
I20250809 02:05:16.334255 26799 raft_consensus.cc:1273] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 4 FOLLOWER]: Refusing update from remote peer f948ab3509494e3b983291c60a264978: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250809 02:05:16.335719 27168 consensus_queue.cc:1035] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250809 02:05:16.340591 27167 raft_consensus.cc:2953] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 96d09e651b9d450b97762273a1c7fc77 (127.25.185.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: true } } unsafe_config_change: true }
I20250809 02:05:16.341849 26799 raft_consensus.cc:2953] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 96d09e651b9d450b97762273a1c7fc77 (127.25.185.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: true } } unsafe_config_change: true }
W20250809 02:05:16.346751 26601 consensus_peers.cc:489] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 -> Peer 96d09e651b9d450b97762273a1c7fc77 (127.25.185.193:42963): Couldn't send request to peer 96d09e651b9d450b97762273a1c7fc77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14e76c181acb4439b9c57e6a94bc98d4. This is attempt 1: this message will repeat every 5th retry.
I20250809 02:05:16.347927 27054 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 14e76c181acb4439b9c57e6a94bc98d4 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250809 02:05:16.351279 27068 catalog_manager.cc:5582] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 reported cstate change: config changed from index 3 to 5, NON_VOTER 96d09e651b9d450b97762273a1c7fc77 (127.25.185.193) added. New cstate: current_term: 4 leader_uuid: "f948ab3509494e3b983291c60a264978" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250809 02:05:16.366339 27054 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 14e76c181acb4439b9c57e6a94bc98d4 on TS 8ce76fbab4c74ad3827a525044801d85 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8ce76fbab4c74ad3827a525044801d85
I20250809 02:05:16.769923 27182 ts_tablet_manager.cc:927] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: Initiating tablet copy from peer f948ab3509494e3b983291c60a264978 (127.25.185.194:44771)
I20250809 02:05:16.772536 27182 tablet_copy_client.cc:323] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: tablet copy: Beginning tablet copy session from remote peer at address 127.25.185.194:44771
I20250809 02:05:16.783319 26686 tablet_copy_service.cc:140] P f948ab3509494e3b983291c60a264978: Received BeginTabletCopySession request for tablet 14e76c181acb4439b9c57e6a94bc98d4 from peer 96d09e651b9d450b97762273a1c7fc77 ({username='slave'} at 127.25.185.193:54867)
I20250809 02:05:16.783901 26686 tablet_copy_service.cc:161] P f948ab3509494e3b983291c60a264978: Beginning new tablet copy session on tablet 14e76c181acb4439b9c57e6a94bc98d4 from peer 96d09e651b9d450b97762273a1c7fc77 at {username='slave'} at 127.25.185.193:54867: session id = 96d09e651b9d450b97762273a1c7fc77-14e76c181acb4439b9c57e6a94bc98d4
I20250809 02:05:16.788975 26686 tablet_copy_source_session.cc:215] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978: Tablet Copy: opened 0 blocks and 1 log segments
I20250809 02:05:16.793907 27182 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14e76c181acb4439b9c57e6a94bc98d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:16.812757 27182 tablet_copy_client.cc:806] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: tablet copy: Starting download of 0 data blocks...
I20250809 02:05:16.813280 27182 tablet_copy_client.cc:670] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: tablet copy: Starting download of 1 WAL segments...
I20250809 02:05:16.816763 27182 tablet_copy_client.cc:538] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250809 02:05:16.822332 27182 tablet_bootstrap.cc:492] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: Bootstrap starting.
I20250809 02:05:16.833900 27182 log.cc:826] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:16.844211 27182 tablet_bootstrap.cc:492] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:05:16.844965 27182 tablet_bootstrap.cc:492] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: Bootstrap complete.
I20250809 02:05:16.845570 27182 ts_tablet_manager.cc:1397] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.004s
I20250809 02:05:16.862157 27182 raft_consensus.cc:357] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: true } } unsafe_config_change: true
I20250809 02:05:16.863109 27182 raft_consensus.cc:738] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 96d09e651b9d450b97762273a1c7fc77, State: Initialized, Role: LEARNER
I20250809 02:05:16.863727 27182 consensus_queue.cc:260] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: true } } unsafe_config_change: true
I20250809 02:05:16.867055 27182 ts_tablet_manager.cc:1428] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77: Time spent starting tablet: real 0.021s user 0.016s sys 0.008s
I20250809 02:05:16.868619 26686 tablet_copy_service.cc:342] P f948ab3509494e3b983291c60a264978: Request end of tablet copy session 96d09e651b9d450b97762273a1c7fc77-14e76c181acb4439b9c57e6a94bc98d4 received from {username='slave'} at 127.25.185.193:54867
I20250809 02:05:16.868963 26686 tablet_copy_service.cc:434] P f948ab3509494e3b983291c60a264978: ending tablet copy session 96d09e651b9d450b97762273a1c7fc77-14e76c181acb4439b9c57e6a94bc98d4 on tablet 14e76c181acb4439b9c57e6a94bc98d4 with peer 96d09e651b9d450b97762273a1c7fc77
I20250809 02:05:17.228597 26533 raft_consensus.cc:1215] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250809 02:05:17.533185 27054 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 14e76c181acb4439b9c57e6a94bc98d4 on TS 8ce76fbab4c74ad3827a525044801d85 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 8ce76fbab4c74ad3827a525044801d85
I20250809 02:05:17.766219 27189 raft_consensus.cc:1062] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978: attempting to promote NON_VOTER 96d09e651b9d450b97762273a1c7fc77 to VOTER
I20250809 02:05:17.767796 27189 consensus_queue.cc:237] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: false } } unsafe_config_change: true
I20250809 02:05:17.772310 26533 raft_consensus.cc:1273] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77 [term 4 LEARNER]: Refusing update from remote peer f948ab3509494e3b983291c60a264978: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250809 02:05:17.773289 26799 raft_consensus.cc:1273] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 4 FOLLOWER]: Refusing update from remote peer f948ab3509494e3b983291c60a264978: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250809 02:05:17.773419 27189 consensus_queue.cc:1035] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250809 02:05:17.774349 27188 consensus_queue.cc:1035] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250809 02:05:17.780535 27191 raft_consensus.cc:2953] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 96d09e651b9d450b97762273a1c7fc77 (127.25.185.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: false } } unsafe_config_change: true }
I20250809 02:05:17.782016 26533 raft_consensus.cc:2953] T 14e76c181acb4439b9c57e6a94bc98d4 P 96d09e651b9d450b97762273a1c7fc77 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 96d09e651b9d450b97762273a1c7fc77 (127.25.185.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: false } } unsafe_config_change: true }
I20250809 02:05:17.782114 26799 raft_consensus.cc:2953] T 14e76c181acb4439b9c57e6a94bc98d4 P 75b03544cc9b4dc1b59832de7b400910 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 96d09e651b9d450b97762273a1c7fc77 (127.25.185.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: false } } unsafe_config_change: true }
I20250809 02:05:17.793524 27068 catalog_manager.cc:5582] T 14e76c181acb4439b9c57e6a94bc98d4 P f948ab3509494e3b983291c60a264978 reported cstate change: config changed from index 5 to 6, 96d09e651b9d450b97762273a1c7fc77 (127.25.185.193) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "f948ab3509494e3b983291c60a264978" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "75b03544cc9b4dc1b59832de7b400910" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 35225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f948ab3509494e3b983291c60a264978" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 44771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "96d09e651b9d450b97762273a1c7fc77" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 42963 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250809 02:05:17.823889 26343 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250809 02:05:17.836270 26343 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "14e76c181acb4439b9c57e6a94bc98d4"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "75b03544cc9b4dc1b59832de7b400910"
rpc_addresses {
host: "127.25.185.195"
port: 35225
}
}
ts_infos {
permanent_uuid: "f948ab3509494e3b983291c60a264978"
rpc_addresses {
host: "127.25.185.194"
port: 44771
}
}
ts_infos {
permanent_uuid: "96d09e651b9d450b97762273a1c7fc77"
rpc_addresses {
host: "127.25.185.193"
port: 42963
}
}
I20250809 02:05:17.838383 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 26449
I20250809 02:05:17.862295 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 26582
I20250809 02:05:17.889801 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 26715
I20250809 02:05:17.917847 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 27020
2025-08-09T02:05:17Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (20143 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250809 02:05:17.989007 26343 test_util.cc:276] Using random seed: 524175480
I20250809 02:05:17.994877 26343 ts_itest-base.cc:115] Starting cluster with:
I20250809 02:05:17.995048 26343 ts_itest-base.cc:116] --------------
I20250809 02:05:17.995160 26343 ts_itest-base.cc:117] 3 tablet servers
I20250809 02:05:17.995260 26343 ts_itest-base.cc:118] 3 replicas per TS
I20250809 02:05:17.995360 26343 ts_itest-base.cc:119] --------------
2025-08-09T02:05:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T02:05:18Z Disabled control of system clock
I20250809 02:05:18.031863 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:36535
--webserver_interface=127.25.185.254
--webserver_port=0
--builtin_ntp_servers=127.25.185.212:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:36535
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250809 02:05:18.327077 27210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:18.327642 27210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:18.328045 27210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:18.358249 27210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:05:18.358579 27210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:18.358793 27210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:05:18.358985 27210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:05:18.393690 27210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:36535
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:36535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:18.394986 27210 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:18.396493 27210 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:18.408329 27216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:18.408648 27217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:19.573210 27218 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250809 02:05:19.574795 27219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:19.574815 27210 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:19.577662 27210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:19.580233 27210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:19.581576 27210 hybrid_clock.cc:648] HybridClock initialized: now 1754705119581534 us; error 54 us; skew 500 ppm
I20250809 02:05:19.582450 27210 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:19.589854 27210 webserver.cc:489] Webserver started at http://127.25.185.254:39845/ using document root <none> and password file <none>
I20250809 02:05:19.590974 27210 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:19.591224 27210 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:19.591742 27210 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:19.596241 27210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "94c7b41bd50b4e4780da785bc97506a0"
format_stamp: "Formatted at 2025-08-09 02:05:19 on dist-test-slave-24ff"
I20250809 02:05:19.597337 27210 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "94c7b41bd50b4e4780da785bc97506a0"
format_stamp: "Formatted at 2025-08-09 02:05:19 on dist-test-slave-24ff"
I20250809 02:05:19.604496 27210 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250809 02:05:19.610275 27226 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:19.611366 27210 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250809 02:05:19.611692 27210 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "94c7b41bd50b4e4780da785bc97506a0"
format_stamp: "Formatted at 2025-08-09 02:05:19 on dist-test-slave-24ff"
I20250809 02:05:19.612037 27210 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:19.666177 27210 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:19.667722 27210 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:19.668160 27210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:19.738122 27210 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:36535
I20250809 02:05:19.738200 27277 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:36535 every 8 connection(s)
I20250809 02:05:19.740844 27210 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 02:05:19.746028 27278 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:19.746335 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 27210
I20250809 02:05:19.746774 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 02:05:19.770895 27278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0: Bootstrap starting.
I20250809 02:05:19.777841 27278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:19.779752 27278 log.cc:826] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:19.784253 27278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0: No bootstrap required, opened a new log
I20250809 02:05:19.802667 27278 raft_consensus.cc:357] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c7b41bd50b4e4780da785bc97506a0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36535 } }
I20250809 02:05:19.803651 27278 raft_consensus.cc:383] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:19.803951 27278 raft_consensus.cc:738] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c7b41bd50b4e4780da785bc97506a0, State: Initialized, Role: FOLLOWER
I20250809 02:05:19.804643 27278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c7b41bd50b4e4780da785bc97506a0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36535 } }
I20250809 02:05:19.805128 27278 raft_consensus.cc:397] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:05:19.805358 27278 raft_consensus.cc:491] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:05:19.805675 27278 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:19.809617 27278 raft_consensus.cc:513] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c7b41bd50b4e4780da785bc97506a0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36535 } }
I20250809 02:05:19.810299 27278 leader_election.cc:304] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94c7b41bd50b4e4780da785bc97506a0; no voters:
I20250809 02:05:19.811921 27278 leader_election.cc:290] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:05:19.812559 27283 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:05:19.815590 27283 raft_consensus.cc:695] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [term 1 LEADER]: Becoming Leader. State: Replica: 94c7b41bd50b4e4780da785bc97506a0, State: Running, Role: LEADER
I20250809 02:05:19.815999 27278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:05:19.816380 27283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c7b41bd50b4e4780da785bc97506a0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36535 } }
I20250809 02:05:19.827108 27285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c7b41bd50b4e4780da785bc97506a0. Latest consensus state: current_term: 1 leader_uuid: "94c7b41bd50b4e4780da785bc97506a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c7b41bd50b4e4780da785bc97506a0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36535 } } }
I20250809 02:05:19.827159 27284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94c7b41bd50b4e4780da785bc97506a0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c7b41bd50b4e4780da785bc97506a0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36535 } } }
I20250809 02:05:19.828078 27284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [sys.catalog]: This master's current role is: LEADER
I20250809 02:05:19.831957 27285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0 [sys.catalog]: This master's current role is: LEADER
I20250809 02:05:19.838088 27292 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:05:19.848474 27292 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:05:19.873639 27292 catalog_manager.cc:1349] Generated new cluster ID: cf0a3e5e553c4312aa8c3804ac49b692
I20250809 02:05:19.873904 27292 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:05:19.895033 27292 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 02:05:19.897044 27292 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:05:19.926148 27292 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 94c7b41bd50b4e4780da785bc97506a0: Generated new TSK 0
I20250809 02:05:19.927099 27292 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 02:05:19.939994 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:0
--local_ip_for_outbound_sockets=127.25.185.193
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36535
--builtin_ntp_servers=127.25.185.212:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250809 02:05:20.260582 27303 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250809 02:05:20.261235 27303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:20.261552 27303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:20.262065 27303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:20.297211 27303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:20.298111 27303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:05:20.336309 27303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:20.337682 27303 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:20.339375 27303 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:20.352679 27309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:21.757596 27308 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 27303
W20250809 02:05:22.389165 27308 kernel_stack_watchdog.cc:198] Thread 27303 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 02:05:22.389561 27303 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.035s user 0.564s sys 1.433s
W20250809 02:05:20.353785 27310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:22.390053 27303 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.036s user 0.564s sys 1.433s
W20250809 02:05:22.392381 27312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:22.396733 27311 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2040 milliseconds
I20250809 02:05:22.396751 27303 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:22.397991 27303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:22.400235 27303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:22.401599 27303 hybrid_clock.cc:648] HybridClock initialized: now 1754705122401533 us; error 66 us; skew 500 ppm
I20250809 02:05:22.402388 27303 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:22.408905 27303 webserver.cc:489] Webserver started at http://127.25.185.193:34777/ using document root <none> and password file <none>
I20250809 02:05:22.409852 27303 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:22.410046 27303 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:22.410491 27303 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:22.415244 27303 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f5f89bf2b24c426f82d955949b7d1264"
format_stamp: "Formatted at 2025-08-09 02:05:22 on dist-test-slave-24ff"
I20250809 02:05:22.416555 27303 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f5f89bf2b24c426f82d955949b7d1264"
format_stamp: "Formatted at 2025-08-09 02:05:22 on dist-test-slave-24ff"
I20250809 02:05:22.444818 27303 fs_manager.cc:696] Time spent creating directory manager: real 0.028s user 0.007s sys 0.003s
I20250809 02:05:22.451097 27319 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:22.452297 27303 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250809 02:05:22.452647 27303 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f5f89bf2b24c426f82d955949b7d1264"
format_stamp: "Formatted at 2025-08-09 02:05:22 on dist-test-slave-24ff"
I20250809 02:05:22.453012 27303 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:22.501261 27303 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:22.502812 27303 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:22.503224 27303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:22.505846 27303 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:22.510792 27303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:22.511006 27303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:22.511319 27303 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:22.511488 27303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:22.688207 27303 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:37929
I20250809 02:05:22.688304 27431 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:37929 every 8 connection(s)
I20250809 02:05:22.690996 27303 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 02:05:22.701069 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 27303
I20250809 02:05:22.701596 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 02:05:22.709193 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.194:0
--local_ip_for_outbound_sockets=127.25.185.194
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36535
--builtin_ntp_servers=127.25.185.212:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250809 02:05:22.717868 27432 heartbeater.cc:344] Connected to a master server at 127.25.185.254:36535
I20250809 02:05:22.718297 27432 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:22.719451 27432 heartbeater.cc:507] Master 127.25.185.254:36535 requested a full tablet report, sending...
I20250809 02:05:22.721786 27243 ts_manager.cc:194] Registered new tserver with Master: f5f89bf2b24c426f82d955949b7d1264 (127.25.185.193:37929)
I20250809 02:05:22.723708 27243 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:35573
W20250809 02:05:23.019201 27436 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250809 02:05:23.019829 27436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:23.020123 27436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:23.020619 27436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:23.051350 27436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:23.052201 27436 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.194
I20250809 02:05:23.086109 27436 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.194
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:23.087412 27436 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:23.089012 27436 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:23.100419 27442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:23.727058 27432 heartbeater.cc:499] Master 127.25.185.254:36535 was elected leader, sending a full tablet report...
W20250809 02:05:24.504074 27441 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 27436
W20250809 02:05:24.828215 27441 kernel_stack_watchdog.cc:198] Thread 27436 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 02:05:23.101159 27443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:24.828910 27436 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.728s user 0.634s sys 1.090s
W20250809 02:05:24.829455 27436 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.729s user 0.634s sys 1.091s
W20250809 02:05:24.831250 27445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:24.834199 27444 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1732 milliseconds
I20250809 02:05:24.834214 27436 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:24.835418 27436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:24.837394 27436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:24.838757 27436 hybrid_clock.cc:648] HybridClock initialized: now 1754705124838704 us; error 49 us; skew 500 ppm
I20250809 02:05:24.839636 27436 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:24.845294 27436 webserver.cc:489] Webserver started at http://127.25.185.194:43523/ using document root <none> and password file <none>
I20250809 02:05:24.846287 27436 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:24.846503 27436 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:24.846972 27436 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:24.851236 27436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0118abdd2a5e4de7b377658f2f3f38d6"
format_stamp: "Formatted at 2025-08-09 02:05:24 on dist-test-slave-24ff"
I20250809 02:05:24.852334 27436 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0118abdd2a5e4de7b377658f2f3f38d6"
format_stamp: "Formatted at 2025-08-09 02:05:24 on dist-test-slave-24ff"
I20250809 02:05:24.859155 27436 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20250809 02:05:24.864660 27452 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:24.865737 27436 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.005s
I20250809 02:05:24.866082 27436 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0118abdd2a5e4de7b377658f2f3f38d6"
format_stamp: "Formatted at 2025-08-09 02:05:24 on dist-test-slave-24ff"
I20250809 02:05:24.866411 27436 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:24.914953 27436 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:24.916419 27436 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:24.916839 27436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:24.919422 27436 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:24.923432 27436 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:24.923659 27436 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:24.923906 27436 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:24.924077 27436 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:25.056453 27436 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.194:41171
I20250809 02:05:25.056558 27564 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.194:41171 every 8 connection(s)
I20250809 02:05:25.059026 27436 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 02:05:25.064030 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 27436
I20250809 02:05:25.064424 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 02:05:25.070487 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.195:0
--local_ip_for_outbound_sockets=127.25.185.195
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36535
--builtin_ntp_servers=127.25.185.212:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250809 02:05:25.091928 27565 heartbeater.cc:344] Connected to a master server at 127.25.185.254:36535
I20250809 02:05:25.092437 27565 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:25.093745 27565 heartbeater.cc:507] Master 127.25.185.254:36535 requested a full tablet report, sending...
I20250809 02:05:25.096164 27243 ts_manager.cc:194] Registered new tserver with Master: 0118abdd2a5e4de7b377658f2f3f38d6 (127.25.185.194:41171)
I20250809 02:05:25.097800 27243 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.194:42527
W20250809 02:05:25.367101 27569 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250809 02:05:25.367692 27569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:25.367899 27569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:25.368304 27569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:25.398906 27569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:25.399783 27569 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.195
I20250809 02:05:25.434588 27569 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36535
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.195
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:25.435861 27569 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:25.437355 27569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:25.448818 27575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:26.101601 27565 heartbeater.cc:499] Master 127.25.185.254:36535 was elected leader, sending a full tablet report...
W20250809 02:05:25.450819 27576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:26.799098 27578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:26.804464 27577 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1353 milliseconds
W20250809 02:05:26.804610 27569 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.355s user 0.473s sys 0.874s
W20250809 02:05:26.804966 27569 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.355s user 0.473s sys 0.874s
I20250809 02:05:26.805212 27569 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:26.806252 27569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:26.808439 27569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:26.809785 27569 hybrid_clock.cc:648] HybridClock initialized: now 1754705126809731 us; error 61 us; skew 500 ppm
I20250809 02:05:26.810566 27569 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:26.818176 27569 webserver.cc:489] Webserver started at http://127.25.185.195:46481/ using document root <none> and password file <none>
I20250809 02:05:26.819245 27569 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:26.819491 27569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:26.819944 27569 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:26.824326 27569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "74bec9fd17b44d54992ec702e0daba3c"
format_stamp: "Formatted at 2025-08-09 02:05:26 on dist-test-slave-24ff"
I20250809 02:05:26.825408 27569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "74bec9fd17b44d54992ec702e0daba3c"
format_stamp: "Formatted at 2025-08-09 02:05:26 on dist-test-slave-24ff"
I20250809 02:05:26.833560 27569 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.001s sys 0.009s
I20250809 02:05:26.839748 27585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:26.840863 27569 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250809 02:05:26.841174 27569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "74bec9fd17b44d54992ec702e0daba3c"
format_stamp: "Formatted at 2025-08-09 02:05:26 on dist-test-slave-24ff"
I20250809 02:05:26.841509 27569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:26.919560 27569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:26.921056 27569 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:26.921468 27569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:26.923913 27569 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:26.927919 27569 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:26.928176 27569 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:26.928411 27569 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:26.928566 27569 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:27.059089 27569 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.195:44755
I20250809 02:05:27.059201 27697 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.195:44755 every 8 connection(s)
I20250809 02:05:27.061832 27569 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 02:05:27.072016 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 27569
I20250809 02:05:27.072587 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 02:05:27.086566 27698 heartbeater.cc:344] Connected to a master server at 127.25.185.254:36535
I20250809 02:05:27.087116 27698 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:27.088128 27698 heartbeater.cc:507] Master 127.25.185.254:36535 requested a full tablet report, sending...
I20250809 02:05:27.090636 27243 ts_manager.cc:194] Registered new tserver with Master: 74bec9fd17b44d54992ec702e0daba3c (127.25.185.195:44755)
I20250809 02:05:27.092128 27243 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.195:38345
I20250809 02:05:27.095687 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:05:27.133962 27243 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50984:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250809 02:05:27.152304 27243 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 02:05:27.206327 27633 tablet_service.cc:1468] Processing CreateTablet for tablet 048e08f2d6b34980837f97a2d2333218 (DEFAULT_TABLE table=TestTable [id=40619f919b434447aa4fb22b1ad5964c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:05:27.208573 27633 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 048e08f2d6b34980837f97a2d2333218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:27.214524 27367 tablet_service.cc:1468] Processing CreateTablet for tablet 048e08f2d6b34980837f97a2d2333218 (DEFAULT_TABLE table=TestTable [id=40619f919b434447aa4fb22b1ad5964c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:05:27.214630 27500 tablet_service.cc:1468] Processing CreateTablet for tablet 048e08f2d6b34980837f97a2d2333218 (DEFAULT_TABLE table=TestTable [id=40619f919b434447aa4fb22b1ad5964c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:05:27.216463 27500 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 048e08f2d6b34980837f97a2d2333218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:27.216576 27367 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 048e08f2d6b34980837f97a2d2333218. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:27.236828 27718 tablet_bootstrap.cc:492] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264: Bootstrap starting.
I20250809 02:05:27.242128 27717 tablet_bootstrap.cc:492] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c: Bootstrap starting.
I20250809 02:05:27.243772 27718 tablet_bootstrap.cc:654] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:27.246301 27719 tablet_bootstrap.cc:492] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6: Bootstrap starting.
I20250809 02:05:27.246333 27718 log.cc:826] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:27.249728 27717 tablet_bootstrap.cc:654] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:27.251564 27717 log.cc:826] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:27.252070 27718 tablet_bootstrap.cc:492] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264: No bootstrap required, opened a new log
I20250809 02:05:27.252606 27718 ts_tablet_manager.cc:1397] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.005s
I20250809 02:05:27.253326 27719 tablet_bootstrap.cc:654] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:27.255620 27719 log.cc:826] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:27.256649 27717 tablet_bootstrap.cc:492] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c: No bootstrap required, opened a new log
I20250809 02:05:27.257061 27717 ts_tablet_manager.cc:1397] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250809 02:05:27.261018 27719 tablet_bootstrap.cc:492] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6: No bootstrap required, opened a new log
I20250809 02:05:27.261514 27719 ts_tablet_manager.cc:1397] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.005s
I20250809 02:05:27.275976 27717 raft_consensus.cc:357] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:27.276711 27717 raft_consensus.cc:738] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74bec9fd17b44d54992ec702e0daba3c, State: Initialized, Role: FOLLOWER
I20250809 02:05:27.277311 27717 consensus_queue.cc:260] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:27.278970 27718 raft_consensus.cc:357] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:27.279889 27718 raft_consensus.cc:738] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5f89bf2b24c426f82d955949b7d1264, State: Initialized, Role: FOLLOWER
I20250809 02:05:27.280331 27698 heartbeater.cc:499] Master 127.25.185.254:36535 was elected leader, sending a full tablet report...
I20250809 02:05:27.280647 27718 consensus_queue.cc:260] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:27.282305 27717 ts_tablet_manager.cc:1428] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c: Time spent starting tablet: real 0.025s user 0.011s sys 0.012s
I20250809 02:05:27.285151 27718 ts_tablet_manager.cc:1428] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264: Time spent starting tablet: real 0.032s user 0.030s sys 0.001s
I20250809 02:05:27.287580 27719 raft_consensus.cc:357] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:27.288290 27719 raft_consensus.cc:738] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0118abdd2a5e4de7b377658f2f3f38d6, State: Initialized, Role: FOLLOWER
I20250809 02:05:27.288872 27719 consensus_queue.cc:260] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:27.293435 27719 ts_tablet_manager.cc:1428] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6: Time spent starting tablet: real 0.032s user 0.029s sys 0.004s
W20250809 02:05:27.315527 27566 tablet.cc:2378] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 02:05:27.316633 27699 tablet.cc:2378] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:05:27.325469 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:05:27.328812 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f5f89bf2b24c426f82d955949b7d1264 to finish bootstrapping
I20250809 02:05:27.341241 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0118abdd2a5e4de7b377658f2f3f38d6 to finish bootstrapping
I20250809 02:05:27.351747 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 74bec9fd17b44d54992ec702e0daba3c to finish bootstrapping
I20250809 02:05:27.392882 27387 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "048e08f2d6b34980837f97a2d2333218"
dest_uuid: "f5f89bf2b24c426f82d955949b7d1264"
from {username='slave'} at 127.0.0.1:44296
I20250809 02:05:27.393440 27387 raft_consensus.cc:491] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250809 02:05:27.393762 27387 raft_consensus.cc:3058] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:27.398049 27387 raft_consensus.cc:513] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:27.400408 27387 leader_election.cc:290] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [CANDIDATE]: Term 1 election: Requested vote from peers 74bec9fd17b44d54992ec702e0daba3c (127.25.185.195:44755), 0118abdd2a5e4de7b377658f2f3f38d6 (127.25.185.194:41171)
I20250809 02:05:27.408361 26343 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250809 02:05:27.412704 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "048e08f2d6b34980837f97a2d2333218" candidate_uuid: "f5f89bf2b24c426f82d955949b7d1264" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "74bec9fd17b44d54992ec702e0daba3c"
I20250809 02:05:27.413129 27520 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "048e08f2d6b34980837f97a2d2333218" candidate_uuid: "f5f89bf2b24c426f82d955949b7d1264" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0118abdd2a5e4de7b377658f2f3f38d6"
I20250809 02:05:27.413503 27653 raft_consensus.cc:3058] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:27.413863 27520 raft_consensus.cc:3058] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:27.419749 27653 raft_consensus.cc:2466] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f89bf2b24c426f82d955949b7d1264 in term 1.
I20250809 02:05:27.421211 27321 leader_election.cc:304] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74bec9fd17b44d54992ec702e0daba3c, f5f89bf2b24c426f82d955949b7d1264; no voters:
I20250809 02:05:27.421870 27520 raft_consensus.cc:2466] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5f89bf2b24c426f82d955949b7d1264 in term 1.
I20250809 02:05:27.422268 27724 raft_consensus.cc:2802] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:05:27.424377 27724 raft_consensus.cc:695] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 1 LEADER]: Becoming Leader. State: Replica: f5f89bf2b24c426f82d955949b7d1264, State: Running, Role: LEADER
I20250809 02:05:27.425225 27724 consensus_queue.cc:237] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:27.436298 27243 catalog_manager.cc:5582] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 reported cstate change: term changed from 0 to 1, leader changed from <none> to f5f89bf2b24c426f82d955949b7d1264 (127.25.185.193). New cstate: current_term: 1 leader_uuid: "f5f89bf2b24c426f82d955949b7d1264" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } health_report { overall_health: HEALTHY } } }
W20250809 02:05:27.451005 27433 tablet.cc:2378] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:05:27.513741 26343 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250809 02:05:27.718576 26343 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250809 02:05:27.826013 27724 consensus_queue.cc:1035] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 02:05:27.841738 27724 consensus_queue.cc:1035] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 02:05:29.678416 27387 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "048e08f2d6b34980837f97a2d2333218"
dest_uuid: "f5f89bf2b24c426f82d955949b7d1264"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44302
I20250809 02:05:29.679097 27387 raft_consensus.cc:604] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 1 LEADER]: Received request to transfer leadership
I20250809 02:05:29.789407 27762 raft_consensus.cc:991] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264: : Instructing follower 0118abdd2a5e4de7b377658f2f3f38d6 to start an election
I20250809 02:05:29.789848 27747 raft_consensus.cc:1079] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 1 LEADER]: Signalling peer 0118abdd2a5e4de7b377658f2f3f38d6 to start an election
I20250809 02:05:29.791213 27520 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "048e08f2d6b34980837f97a2d2333218"
dest_uuid: "0118abdd2a5e4de7b377658f2f3f38d6"
from {username='slave'} at 127.25.185.193:51081
I20250809 02:05:29.791733 27520 raft_consensus.cc:491] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250809 02:05:29.792006 27520 raft_consensus.cc:3058] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:05:29.796097 27520 raft_consensus.cc:513] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:29.798296 27520 leader_election.cc:290] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [CANDIDATE]: Term 2 election: Requested vote from peers 74bec9fd17b44d54992ec702e0daba3c (127.25.185.195:44755), f5f89bf2b24c426f82d955949b7d1264 (127.25.185.193:37929)
I20250809 02:05:29.809017 27653 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "048e08f2d6b34980837f97a2d2333218" candidate_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "74bec9fd17b44d54992ec702e0daba3c"
I20250809 02:05:29.809581 27653 raft_consensus.cc:3058] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:05:29.810142 27387 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "048e08f2d6b34980837f97a2d2333218" candidate_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "f5f89bf2b24c426f82d955949b7d1264"
I20250809 02:05:29.810797 27387 raft_consensus.cc:3053] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 1 LEADER]: Stepping down as leader of term 1
I20250809 02:05:29.811116 27387 raft_consensus.cc:738] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: f5f89bf2b24c426f82d955949b7d1264, State: Running, Role: LEADER
I20250809 02:05:29.811751 27387 consensus_queue.cc:260] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:29.812988 27387 raft_consensus.cc:3058] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:05:29.815909 27653 raft_consensus.cc:2466] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0118abdd2a5e4de7b377658f2f3f38d6 in term 2.
I20250809 02:05:29.817142 27454 leader_election.cc:304] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0118abdd2a5e4de7b377658f2f3f38d6, 74bec9fd17b44d54992ec702e0daba3c; no voters:
I20250809 02:05:29.819177 27766 raft_consensus.cc:2802] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 02:05:29.819375 27387 raft_consensus.cc:2466] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0118abdd2a5e4de7b377658f2f3f38d6 in term 2.
I20250809 02:05:29.820964 27766 raft_consensus.cc:695] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [term 2 LEADER]: Becoming Leader. State: Replica: 0118abdd2a5e4de7b377658f2f3f38d6, State: Running, Role: LEADER
I20250809 02:05:29.821769 27766 consensus_queue.cc:237] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } }
I20250809 02:05:29.829056 27242 catalog_manager.cc:5582] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 reported cstate change: term changed from 1 to 2, leader changed from f5f89bf2b24c426f82d955949b7d1264 (127.25.185.193) to 0118abdd2a5e4de7b377658f2f3f38d6 (127.25.185.194). New cstate: current_term: 2 leader_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0118abdd2a5e4de7b377658f2f3f38d6" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41171 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 } health_report { overall_health: UNKNOWN } } }
I20250809 02:05:30.315908 27387 raft_consensus.cc:1273] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 2 FOLLOWER]: Refusing update from remote peer 0118abdd2a5e4de7b377658f2f3f38d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250809 02:05:30.317437 27766 consensus_queue.cc:1035] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5f89bf2b24c426f82d955949b7d1264" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250809 02:05:30.329819 27653 raft_consensus.cc:1273] T 048e08f2d6b34980837f97a2d2333218 P 74bec9fd17b44d54992ec702e0daba3c [term 2 FOLLOWER]: Refusing update from remote peer 0118abdd2a5e4de7b377658f2f3f38d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250809 02:05:30.331498 27778 consensus_queue.cc:1035] T 048e08f2d6b34980837f97a2d2333218 P 0118abdd2a5e4de7b377658f2f3f38d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74bec9fd17b44d54992ec702e0daba3c" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 44755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250809 02:05:32.366262 27387 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "048e08f2d6b34980837f97a2d2333218"
dest_uuid: "f5f89bf2b24c426f82d955949b7d1264"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:44312
I20250809 02:05:32.367007 27387 raft_consensus.cc:604] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 2 FOLLOWER]: Received request to transfer leadership
I20250809 02:05:32.367363 27387 raft_consensus.cc:612] T 048e08f2d6b34980837f97a2d2333218 P f5f89bf2b24c426f82d955949b7d1264 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250809 02:05:33.402434 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 27303
I20250809 02:05:33.427198 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 27436
I20250809 02:05:33.452545 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 27569
I20250809 02:05:33.477594 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 27210
2025-08-09T02:05:33Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (15542 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250809 02:05:33.531927 26343 test_util.cc:276] Using random seed: 539718379
I20250809 02:05:33.536379 26343 ts_itest-base.cc:115] Starting cluster with:
I20250809 02:05:33.536548 26343 ts_itest-base.cc:116] --------------
I20250809 02:05:33.536703 26343 ts_itest-base.cc:117] 3 tablet servers
I20250809 02:05:33.536835 26343 ts_itest-base.cc:118] 3 replicas per TS
I20250809 02:05:33.536971 26343 ts_itest-base.cc:119] --------------
2025-08-09T02:05:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T02:05:33Z Disabled control of system clock
I20250809 02:05:33.573871 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:38631
--webserver_interface=127.25.185.254
--webserver_port=0
--builtin_ntp_servers=127.25.185.212:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:38631 with env {}
W20250809 02:05:33.871832 27814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:33.872437 27814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:33.872900 27814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:33.903864 27814 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:05:33.904207 27814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:33.904496 27814 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:05:33.904742 27814 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:05:33.939680 27814 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:38631
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:38631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:33.940996 27814 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:33.942603 27814 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:33.954267 27820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:33.955431 27821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:35.155921 27823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:35.158353 27822 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1198 milliseconds
I20250809 02:05:35.158443 27814 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:35.159734 27814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:35.162796 27814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:35.164153 27814 hybrid_clock.cc:648] HybridClock initialized: now 1754705135164117 us; error 39 us; skew 500 ppm
I20250809 02:05:35.164945 27814 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:35.171403 27814 webserver.cc:489] Webserver started at http://127.25.185.254:38521/ using document root <none> and password file <none>
I20250809 02:05:35.172338 27814 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:35.172566 27814 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:35.173017 27814 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:35.177261 27814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8ee95ea0b5814c4a899c92dbfb657205"
format_stamp: "Formatted at 2025-08-09 02:05:35 on dist-test-slave-24ff"
I20250809 02:05:35.178362 27814 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8ee95ea0b5814c4a899c92dbfb657205"
format_stamp: "Formatted at 2025-08-09 02:05:35 on dist-test-slave-24ff"
I20250809 02:05:35.185557 27814 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250809 02:05:35.191231 27830 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:35.192384 27814 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250809 02:05:35.192740 27814 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8ee95ea0b5814c4a899c92dbfb657205"
format_stamp: "Formatted at 2025-08-09 02:05:35 on dist-test-slave-24ff"
I20250809 02:05:35.193094 27814 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:35.246372 27814 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:35.247946 27814 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:35.248381 27814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:35.317680 27814 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:38631
I20250809 02:05:35.317750 27881 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:38631 every 8 connection(s)
I20250809 02:05:35.320369 27814 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 02:05:35.324736 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 27814
I20250809 02:05:35.325089 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 02:05:35.325385 27882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:35.349201 27882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205: Bootstrap starting.
I20250809 02:05:35.354290 27882 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:35.356215 27882 log.cc:826] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:35.360705 27882 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205: No bootstrap required, opened a new log
I20250809 02:05:35.377350 27882 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee95ea0b5814c4a899c92dbfb657205" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38631 } }
I20250809 02:05:35.378017 27882 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:35.378269 27882 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee95ea0b5814c4a899c92dbfb657205, State: Initialized, Role: FOLLOWER
I20250809 02:05:35.378971 27882 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee95ea0b5814c4a899c92dbfb657205" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38631 } }
I20250809 02:05:35.379483 27882 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:05:35.379750 27882 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:05:35.380052 27882 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:35.384042 27882 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee95ea0b5814c4a899c92dbfb657205" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38631 } }
I20250809 02:05:35.384701 27882 leader_election.cc:304] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ee95ea0b5814c4a899c92dbfb657205; no voters:
I20250809 02:05:35.386303 27882 leader_election.cc:290] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:05:35.387049 27887 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:05:35.389338 27887 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee95ea0b5814c4a899c92dbfb657205, State: Running, Role: LEADER
I20250809 02:05:35.390065 27887 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee95ea0b5814c4a899c92dbfb657205" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38631 } }
I20250809 02:05:35.391136 27882 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:05:35.400389 27889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ee95ea0b5814c4a899c92dbfb657205. Latest consensus state: current_term: 1 leader_uuid: "8ee95ea0b5814c4a899c92dbfb657205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee95ea0b5814c4a899c92dbfb657205" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38631 } } }
I20250809 02:05:35.401446 27889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [sys.catalog]: This master's current role is: LEADER
I20250809 02:05:35.403105 27888 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ee95ea0b5814c4a899c92dbfb657205" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ee95ea0b5814c4a899c92dbfb657205" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38631 } } }
I20250809 02:05:35.403702 27888 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205 [sys.catalog]: This master's current role is: LEADER
I20250809 02:05:35.410351 27897 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:05:35.421275 27897 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:05:35.437909 27897 catalog_manager.cc:1349] Generated new cluster ID: 43182060726e4430b55dc9721590667d
I20250809 02:05:35.438161 27897 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:05:35.461963 27897 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 02:05:35.463444 27897 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:05:35.478786 27897 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8ee95ea0b5814c4a899c92dbfb657205: Generated new TSK 0
I20250809 02:05:35.479671 27897 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 02:05:35.491564 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:0
--local_ip_for_outbound_sockets=127.25.185.193
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38631
--builtin_ntp_servers=127.25.185.212:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250809 02:05:35.806926 27906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:35.807456 27906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:35.807933 27906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:35.839272 27906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:35.840070 27906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:05:35.875748 27906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38631
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:35.877017 27906 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:35.878567 27906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:35.891008 27912 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:35.895117 27915 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:35.892505 27913 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:37.339080 27914 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1445 milliseconds
I20250809 02:05:37.339211 27906 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:37.340768 27906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:37.343523 27906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:37.345000 27906 hybrid_clock.cc:648] HybridClock initialized: now 1754705137344948 us; error 46 us; skew 500 ppm
I20250809 02:05:37.346099 27906 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:37.353782 27906 webserver.cc:489] Webserver started at http://127.25.185.193:36805/ using document root <none> and password file <none>
I20250809 02:05:37.355082 27906 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:37.355412 27906 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:37.356005 27906 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:37.362464 27906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "66996ff260c44f9e8d6cc614628356ca"
format_stamp: "Formatted at 2025-08-09 02:05:37 on dist-test-slave-24ff"
I20250809 02:05:37.364034 27906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "66996ff260c44f9e8d6cc614628356ca"
format_stamp: "Formatted at 2025-08-09 02:05:37 on dist-test-slave-24ff"
I20250809 02:05:37.373823 27906 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20250809 02:05:37.381413 27922 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:37.382674 27906 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250809 02:05:37.383138 27906 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "66996ff260c44f9e8d6cc614628356ca"
format_stamp: "Formatted at 2025-08-09 02:05:37 on dist-test-slave-24ff"
I20250809 02:05:37.383584 27906 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:37.456161 27906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:37.457654 27906 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:37.458088 27906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:37.460707 27906 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:37.464838 27906 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:37.465056 27906 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:37.465289 27906 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:37.465445 27906 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:37.623821 27906 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:45421
I20250809 02:05:37.623986 28034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:45421 every 8 connection(s)
I20250809 02:05:37.626483 27906 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 02:05:37.628894 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 27906
I20250809 02:05:37.629381 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 02:05:37.637694 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.194:0
--local_ip_for_outbound_sockets=127.25.185.194
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38631
--builtin_ntp_servers=127.25.185.212:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:05:37.657742 28035 heartbeater.cc:344] Connected to a master server at 127.25.185.254:38631
I20250809 02:05:37.658288 28035 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:37.659706 28035 heartbeater.cc:507] Master 127.25.185.254:38631 requested a full tablet report, sending...
I20250809 02:05:37.663101 27847 ts_manager.cc:194] Registered new tserver with Master: 66996ff260c44f9e8d6cc614628356ca (127.25.185.193:45421)
I20250809 02:05:37.665844 27847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:35407
W20250809 02:05:37.950721 28039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:37.951423 28039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:37.952232 28039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:37.983424 28039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:37.984287 28039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.194
I20250809 02:05:38.019925 28039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38631
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.194
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:38.021292 28039 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:38.022867 28039 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:38.036345 28046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:38.670248 28035 heartbeater.cc:499] Master 127.25.185.254:38631 was elected leader, sending a full tablet report...
W20250809 02:05:38.037473 28047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:39.280262 28049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:39.283418 28048 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1245 milliseconds
W20250809 02:05:39.284504 28039 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.249s user 0.421s sys 0.817s
W20250809 02:05:39.284754 28039 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.249s user 0.421s sys 0.817s
I20250809 02:05:39.284981 28039 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:39.286046 28039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:39.288471 28039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:39.289835 28039 hybrid_clock.cc:648] HybridClock initialized: now 1754705139289806 us; error 44 us; skew 500 ppm
I20250809 02:05:39.290633 28039 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:39.298477 28039 webserver.cc:489] Webserver started at http://127.25.185.194:33197/ using document root <none> and password file <none>
I20250809 02:05:39.299649 28039 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:39.299903 28039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:39.300388 28039 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:39.305014 28039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "24ea19419fc546c0b6c1ab201b041099"
format_stamp: "Formatted at 2025-08-09 02:05:39 on dist-test-slave-24ff"
I20250809 02:05:39.306182 28039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "24ea19419fc546c0b6c1ab201b041099"
format_stamp: "Formatted at 2025-08-09 02:05:39 on dist-test-slave-24ff"
I20250809 02:05:39.314072 28039 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250809 02:05:39.320492 28056 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:39.321619 28039 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250809 02:05:39.321946 28039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "24ea19419fc546c0b6c1ab201b041099"
format_stamp: "Formatted at 2025-08-09 02:05:39 on dist-test-slave-24ff"
I20250809 02:05:39.322296 28039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:39.401455 28039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:39.403240 28039 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:39.403702 28039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:39.406611 28039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:39.410801 28039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:39.411000 28039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:39.411267 28039 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:39.411427 28039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:39.542341 28039 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.194:38153
I20250809 02:05:39.542426 28168 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.194:38153 every 8 connection(s)
I20250809 02:05:39.544883 28039 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 02:05:39.548588 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 28039
I20250809 02:05:39.549242 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 02:05:39.556602 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.195:0
--local_ip_for_outbound_sockets=127.25.185.195
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38631
--builtin_ntp_servers=127.25.185.212:42455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:05:39.566670 28169 heartbeater.cc:344] Connected to a master server at 127.25.185.254:38631
I20250809 02:05:39.567267 28169 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:39.568610 28169 heartbeater.cc:507] Master 127.25.185.254:38631 requested a full tablet report, sending...
I20250809 02:05:39.571324 27847 ts_manager.cc:194] Registered new tserver with Master: 24ea19419fc546c0b6c1ab201b041099 (127.25.185.194:38153)
I20250809 02:05:39.572666 27847 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.194:49011
W20250809 02:05:39.862967 28173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:39.863485 28173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:39.864064 28173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:39.895388 28173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:39.896255 28173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.195
I20250809 02:05:39.931914 28173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:42455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38631
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.195
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:39.933213 28173 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:39.934904 28173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:39.945976 28179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:40.576706 28169 heartbeater.cc:499] Master 127.25.185.254:38631 was elected leader, sending a full tablet report...
W20250809 02:05:41.350075 28178 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 28173
W20250809 02:05:41.688027 28173 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.741s user 0.566s sys 1.174s
W20250809 02:05:39.948045 28180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:41.688493 28173 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.741s user 0.566s sys 1.175s
W20250809 02:05:41.690708 28182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:41.697060 28181 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1745 milliseconds
I20250809 02:05:41.697141 28173 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:41.698295 28173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:41.700340 28173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:41.701673 28173 hybrid_clock.cc:648] HybridClock initialized: now 1754705141701625 us; error 43 us; skew 500 ppm
I20250809 02:05:41.702445 28173 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:41.708348 28173 webserver.cc:489] Webserver started at http://127.25.185.195:38739/ using document root <none> and password file <none>
I20250809 02:05:41.709264 28173 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:41.709493 28173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:41.709945 28173 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:41.714138 28173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "e383f1727e974b9d86c04f805e2e22d9"
format_stamp: "Formatted at 2025-08-09 02:05:41 on dist-test-slave-24ff"
I20250809 02:05:41.715245 28173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "e383f1727e974b9d86c04f805e2e22d9"
format_stamp: "Formatted at 2025-08-09 02:05:41 on dist-test-slave-24ff"
I20250809 02:05:41.722079 28173 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250809 02:05:41.727674 28189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:41.728724 28173 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250809 02:05:41.729027 28173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "e383f1727e974b9d86c04f805e2e22d9"
format_stamp: "Formatted at 2025-08-09 02:05:41 on dist-test-slave-24ff"
I20250809 02:05:41.729359 28173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:41.792768 28173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:41.794185 28173 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:41.794605 28173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:41.797092 28173 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:41.801121 28173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:41.801322 28173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:41.801590 28173 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:41.801750 28173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:41.931016 28173 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.195:34101
I20250809 02:05:41.931124 28301 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.195:34101 every 8 connection(s)
I20250809 02:05:41.933555 28173 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 02:05:41.936456 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 28173
I20250809 02:05:41.936940 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 02:05:41.956292 28302 heartbeater.cc:344] Connected to a master server at 127.25.185.254:38631
I20250809 02:05:41.956769 28302 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:41.957820 28302 heartbeater.cc:507] Master 127.25.185.254:38631 requested a full tablet report, sending...
I20250809 02:05:41.959834 27846 ts_manager.cc:194] Registered new tserver with Master: e383f1727e974b9d86c04f805e2e22d9 (127.25.185.195:34101)
I20250809 02:05:41.961079 27846 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.195:51321
I20250809 02:05:41.970757 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:05:42.003408 27846 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48910:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250809 02:05:42.022145 27846 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 02:05:42.083312 28237 tablet_service.cc:1468] Processing CreateTablet for tablet ceddc1273fe845ebb8608edc4de7550a (DEFAULT_TABLE table=TestTable [id=79c02c2d946a4972a8d53205bb001624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:05:42.083384 28104 tablet_service.cc:1468] Processing CreateTablet for tablet ceddc1273fe845ebb8608edc4de7550a (DEFAULT_TABLE table=TestTable [id=79c02c2d946a4972a8d53205bb001624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:05:42.085085 28104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceddc1273fe845ebb8608edc4de7550a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:42.085753 28237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceddc1273fe845ebb8608edc4de7550a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:42.086889 27970 tablet_service.cc:1468] Processing CreateTablet for tablet ceddc1273fe845ebb8608edc4de7550a (DEFAULT_TABLE table=TestTable [id=79c02c2d946a4972a8d53205bb001624]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:05:42.088717 27970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ceddc1273fe845ebb8608edc4de7550a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:42.107893 28321 tablet_bootstrap.cc:492] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099: Bootstrap starting.
I20250809 02:05:42.113692 28322 tablet_bootstrap.cc:492] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9: Bootstrap starting.
I20250809 02:05:42.116050 28323 tablet_bootstrap.cc:492] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca: Bootstrap starting.
I20250809 02:05:42.116707 28321 tablet_bootstrap.cc:654] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:42.119221 28321 log.cc:826] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:42.121697 28322 tablet_bootstrap.cc:654] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:42.121814 28323 tablet_bootstrap.cc:654] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:42.124033 28322 log.cc:826] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:42.124148 28323 log.cc:826] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:42.125196 28321 tablet_bootstrap.cc:492] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099: No bootstrap required, opened a new log
I20250809 02:05:42.125801 28321 ts_tablet_manager.cc:1397] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.000s
I20250809 02:05:42.130069 28323 tablet_bootstrap.cc:492] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca: No bootstrap required, opened a new log
I20250809 02:05:42.130358 28322 tablet_bootstrap.cc:492] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9: No bootstrap required, opened a new log
I20250809 02:05:42.130528 28323 ts_tablet_manager.cc:1397] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.002s
I20250809 02:05:42.130879 28322 ts_tablet_manager.cc:1397] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9: Time spent bootstrapping tablet: real 0.018s user 0.012s sys 0.004s
I20250809 02:05:42.149351 28323 raft_consensus.cc:357] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } }
I20250809 02:05:42.150081 28323 raft_consensus.cc:383] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:42.150363 28323 raft_consensus.cc:738] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66996ff260c44f9e8d6cc614628356ca, State: Initialized, Role: FOLLOWER
I20250809 02:05:42.151188 28323 consensus_queue.cc:260] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } }
I20250809 02:05:42.153630 28321 raft_consensus.cc:357] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } }
I20250809 02:05:42.154618 28321 raft_consensus.cc:383] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:42.154991 28321 raft_consensus.cc:738] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ea19419fc546c0b6c1ab201b041099, State: Initialized, Role: FOLLOWER
I20250809 02:05:42.156061 28321 consensus_queue.cc:260] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } }
I20250809 02:05:42.158025 28323 ts_tablet_manager.cc:1428] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca: Time spent starting tablet: real 0.027s user 0.023s sys 0.001s
I20250809 02:05:42.157624 28322 raft_consensus.cc:357] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } }
I20250809 02:05:42.158820 28322 raft_consensus.cc:383] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:42.159245 28322 raft_consensus.cc:738] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e383f1727e974b9d86c04f805e2e22d9, State: Initialized, Role: FOLLOWER
I20250809 02:05:42.160240 28322 consensus_queue.cc:260] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } }
I20250809 02:05:42.160979 28321 ts_tablet_manager.cc:1428] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099: Time spent starting tablet: real 0.035s user 0.026s sys 0.007s
I20250809 02:05:42.163962 28302 heartbeater.cc:499] Master 127.25.185.254:38631 was elected leader, sending a full tablet report...
I20250809 02:05:42.165403 28322 ts_tablet_manager.cc:1428] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9: Time spent starting tablet: real 0.034s user 0.026s sys 0.006s
W20250809 02:05:42.187849 28303 tablet.cc:2378] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:05:42.243727 28327 raft_consensus.cc:491] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:05:42.244226 28327 raft_consensus.cc:513] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } }
I20250809 02:05:42.246616 28327 leader_election.cc:290] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e383f1727e974b9d86c04f805e2e22d9 (127.25.185.195:34101), 24ea19419fc546c0b6c1ab201b041099 (127.25.185.194:38153)
I20250809 02:05:42.257855 28257 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ceddc1273fe845ebb8608edc4de7550a" candidate_uuid: "66996ff260c44f9e8d6cc614628356ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e383f1727e974b9d86c04f805e2e22d9" is_pre_election: true
I20250809 02:05:42.257957 28124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ceddc1273fe845ebb8608edc4de7550a" candidate_uuid: "66996ff260c44f9e8d6cc614628356ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ea19419fc546c0b6c1ab201b041099" is_pre_election: true
I20250809 02:05:42.258828 28124 raft_consensus.cc:2466] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66996ff260c44f9e8d6cc614628356ca in term 0.
I20250809 02:05:42.258828 28257 raft_consensus.cc:2466] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66996ff260c44f9e8d6cc614628356ca in term 0.
I20250809 02:05:42.260116 27925 leader_election.cc:304] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66996ff260c44f9e8d6cc614628356ca, e383f1727e974b9d86c04f805e2e22d9; no voters:
I20250809 02:05:42.260794 28327 raft_consensus.cc:2802] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 02:05:42.261127 28327 raft_consensus.cc:491] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 02:05:42.261364 28327 raft_consensus.cc:3058] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:42.265583 28327 raft_consensus.cc:513] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } }
I20250809 02:05:42.267030 28327 leader_election.cc:290] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [CANDIDATE]: Term 1 election: Requested vote from peers e383f1727e974b9d86c04f805e2e22d9 (127.25.185.195:34101), 24ea19419fc546c0b6c1ab201b041099 (127.25.185.194:38153)
I20250809 02:05:42.267736 28257 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ceddc1273fe845ebb8608edc4de7550a" candidate_uuid: "66996ff260c44f9e8d6cc614628356ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e383f1727e974b9d86c04f805e2e22d9"
I20250809 02:05:42.268023 28124 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ceddc1273fe845ebb8608edc4de7550a" candidate_uuid: "66996ff260c44f9e8d6cc614628356ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24ea19419fc546c0b6c1ab201b041099"
I20250809 02:05:42.268251 28257 raft_consensus.cc:3058] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:42.268513 28124 raft_consensus.cc:3058] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:42.275347 28257 raft_consensus.cc:2466] T ceddc1273fe845ebb8608edc4de7550a P e383f1727e974b9d86c04f805e2e22d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66996ff260c44f9e8d6cc614628356ca in term 1.
I20250809 02:05:42.275331 28124 raft_consensus.cc:2466] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66996ff260c44f9e8d6cc614628356ca in term 1.
I20250809 02:05:42.276367 27925 leader_election.cc:304] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66996ff260c44f9e8d6cc614628356ca, e383f1727e974b9d86c04f805e2e22d9; no voters:
I20250809 02:05:42.276994 28327 raft_consensus.cc:2802] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:05:42.278507 28327 raft_consensus.cc:695] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [term 1 LEADER]: Becoming Leader. State: Replica: 66996ff260c44f9e8d6cc614628356ca, State: Running, Role: LEADER
I20250809 02:05:42.279290 28327 consensus_queue.cc:237] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } }
I20250809 02:05:42.290356 27845 catalog_manager.cc:5582] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 66996ff260c44f9e8d6cc614628356ca (127.25.185.193). New cstate: current_term: 1 leader_uuid: "66996ff260c44f9e8d6cc614628356ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } health_report { overall_health: HEALTHY } } }
W20250809 02:05:42.302326 28170 tablet.cc:2378] T ceddc1273fe845ebb8608edc4de7550a P 24ea19419fc546c0b6c1ab201b041099: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:05:42.327631 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:05:42.331430 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 66996ff260c44f9e8d6cc614628356ca to finish bootstrapping
I20250809 02:05:42.346101 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 24ea19419fc546c0b6c1ab201b041099 to finish bootstrapping
I20250809 02:05:42.357187 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e383f1727e974b9d86c04f805e2e22d9 to finish bootstrapping
I20250809 02:05:42.371290 27845 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48910:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250809 02:05:42.372737 27845 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 02:05:42.388355 27970 tablet_service.cc:1468] Processing CreateTablet for tablet 7a5c4348d74b4e4780bc8eacc2d41e06 (DEFAULT_TABLE table=TestAnotherTable [id=a33d1925d3364661965cc26cab7b92ae]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250809 02:05:42.388839 28104 tablet_service.cc:1468] Processing CreateTablet for tablet 7a5c4348d74b4e4780bc8eacc2d41e06 (DEFAULT_TABLE table=TestAnotherTable [id=a33d1925d3364661965cc26cab7b92ae]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250809 02:05:42.389410 27970 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5c4348d74b4e4780bc8eacc2d41e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:42.389406 28237 tablet_service.cc:1468] Processing CreateTablet for tablet 7a5c4348d74b4e4780bc8eacc2d41e06 (DEFAULT_TABLE table=TestAnotherTable [id=a33d1925d3364661965cc26cab7b92ae]), partition=RANGE (foo) PARTITION UNBOUNDED
W20250809 02:05:42.389611 28036 tablet.cc:2378] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:05:42.389889 28104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5c4348d74b4e4780bc8eacc2d41e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:42.390391 28237 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a5c4348d74b4e4780bc8eacc2d41e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:42.400766 28321 tablet_bootstrap.cc:492] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099: Bootstrap starting.
I20250809 02:05:42.403980 28323 tablet_bootstrap.cc:492] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca: Bootstrap starting.
I20250809 02:05:42.404196 28322 tablet_bootstrap.cc:492] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9: Bootstrap starting.
I20250809 02:05:42.407266 28321 tablet_bootstrap.cc:654] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:42.409281 28323 tablet_bootstrap.cc:654] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:42.410421 28322 tablet_bootstrap.cc:654] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:42.417289 28323 tablet_bootstrap.cc:492] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca: No bootstrap required, opened a new log
I20250809 02:05:42.417398 28321 tablet_bootstrap.cc:492] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099: No bootstrap required, opened a new log
I20250809 02:05:42.417649 28323 ts_tablet_manager.cc:1397] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20250809 02:05:42.417768 28322 tablet_bootstrap.cc:492] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9: No bootstrap required, opened a new log
I20250809 02:05:42.417801 28321 ts_tablet_manager.cc:1397] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.008s
I20250809 02:05:42.418164 28322 ts_tablet_manager.cc:1397] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20250809 02:05:42.419749 28323 raft_consensus.cc:357] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } }
I20250809 02:05:42.420424 28323 raft_consensus.cc:383] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:42.420341 28321 raft_consensus.cc:357] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } }
I20250809 02:05:42.420724 28323 raft_consensus.cc:738] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66996ff260c44f9e8d6cc614628356ca, State: Initialized, Role: FOLLOWER
I20250809 02:05:42.421036 28321 raft_consensus.cc:383] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:42.420619 28322 raft_consensus.cc:357] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } }
I20250809 02:05:42.421319 28321 raft_consensus.cc:738] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24ea19419fc546c0b6c1ab201b041099, State: Initialized, Role: FOLLOWER
I20250809 02:05:42.421401 28322 raft_consensus.cc:383] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:42.421711 28322 raft_consensus.cc:738] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e383f1727e974b9d86c04f805e2e22d9, State: Initialized, Role: FOLLOWER
I20250809 02:05:42.421360 28323 consensus_queue.cc:260] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } }
I20250809 02:05:42.422237 28321 consensus_queue.cc:260] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } }
I20250809 02:05:42.423573 28323 ts_tablet_manager.cc:1428] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca: Time spent starting tablet: real 0.006s user 0.001s sys 0.006s
I20250809 02:05:42.424335 28321 ts_tablet_manager.cc:1428] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250809 02:05:42.422518 28322 consensus_queue.cc:260] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } }
I20250809 02:05:42.426324 28322 ts_tablet_manager.cc:1428] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9: Time spent starting tablet: real 0.008s user 0.001s sys 0.005s
I20250809 02:05:42.505618 28328 raft_consensus.cc:491] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:05:42.506053 28328 raft_consensus.cc:513] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } }
I20250809 02:05:42.508451 28328 leader_election.cc:290] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66996ff260c44f9e8d6cc614628356ca (127.25.185.193:45421), e383f1727e974b9d86c04f805e2e22d9 (127.25.185.195:34101)
I20250809 02:05:42.520599 28257 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a5c4348d74b4e4780bc8eacc2d41e06" candidate_uuid: "24ea19419fc546c0b6c1ab201b041099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e383f1727e974b9d86c04f805e2e22d9" is_pre_election: true
I20250809 02:05:42.520802 27990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a5c4348d74b4e4780bc8eacc2d41e06" candidate_uuid: "24ea19419fc546c0b6c1ab201b041099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66996ff260c44f9e8d6cc614628356ca" is_pre_election: true
I20250809 02:05:42.521061 28257 raft_consensus.cc:2466] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ea19419fc546c0b6c1ab201b041099 in term 0.
I20250809 02:05:42.521425 27990 raft_consensus.cc:2466] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24ea19419fc546c0b6c1ab201b041099 in term 0.
I20250809 02:05:42.522001 28059 leader_election.cc:304] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ea19419fc546c0b6c1ab201b041099, e383f1727e974b9d86c04f805e2e22d9; no voters:
I20250809 02:05:42.522773 28328 raft_consensus.cc:2802] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 02:05:42.523008 28328 raft_consensus.cc:491] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 02:05:42.523200 28328 raft_consensus.cc:3058] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:42.527657 28328 raft_consensus.cc:513] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } }
I20250809 02:05:42.529356 28328 leader_election.cc:290] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [CANDIDATE]: Term 1 election: Requested vote from peers 66996ff260c44f9e8d6cc614628356ca (127.25.185.193:45421), e383f1727e974b9d86c04f805e2e22d9 (127.25.185.195:34101)
I20250809 02:05:42.530023 27990 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a5c4348d74b4e4780bc8eacc2d41e06" candidate_uuid: "24ea19419fc546c0b6c1ab201b041099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66996ff260c44f9e8d6cc614628356ca"
I20250809 02:05:42.530419 27990 raft_consensus.cc:3058] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:42.530287 28257 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a5c4348d74b4e4780bc8eacc2d41e06" candidate_uuid: "24ea19419fc546c0b6c1ab201b041099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e383f1727e974b9d86c04f805e2e22d9"
I20250809 02:05:42.530759 28257 raft_consensus.cc:3058] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:42.534801 28257 raft_consensus.cc:2466] T 7a5c4348d74b4e4780bc8eacc2d41e06 P e383f1727e974b9d86c04f805e2e22d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ea19419fc546c0b6c1ab201b041099 in term 1.
I20250809 02:05:42.534809 27990 raft_consensus.cc:2466] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 66996ff260c44f9e8d6cc614628356ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24ea19419fc546c0b6c1ab201b041099 in term 1.
I20250809 02:05:42.535689 28059 leader_election.cc:304] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24ea19419fc546c0b6c1ab201b041099, e383f1727e974b9d86c04f805e2e22d9; no voters:
I20250809 02:05:42.536321 28328 raft_consensus.cc:2802] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:05:42.537811 28328 raft_consensus.cc:695] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [term 1 LEADER]: Becoming Leader. State: Replica: 24ea19419fc546c0b6c1ab201b041099, State: Running, Role: LEADER
I20250809 02:05:42.538487 28328 consensus_queue.cc:237] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } } peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } }
I20250809 02:05:42.550133 27845 catalog_manager.cc:5582] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24ea19419fc546c0b6c1ab201b041099 (127.25.185.194). New cstate: current_term: 1 leader_uuid: "24ea19419fc546c0b6c1ab201b041099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 } health_report { overall_health: UNKNOWN } } }
I20250809 02:05:42.692462 28327 consensus_queue.cc:1035] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 02:05:42.710450 28327 consensus_queue.cc:1035] T ceddc1273fe845ebb8608edc4de7550a P 66996ff260c44f9e8d6cc614628356ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "24ea19419fc546c0b6c1ab201b041099" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250809 02:05:42.879673 28342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:42.880219 28342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:42.910187 28342 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250809 02:05:42.983984 28328 consensus_queue.cc:1035] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66996ff260c44f9e8d6cc614628356ca" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 45421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250809 02:05:43.011317 28328 consensus_queue.cc:1035] T 7a5c4348d74b4e4780bc8eacc2d41e06 P 24ea19419fc546c0b6c1ab201b041099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e383f1727e974b9d86c04f805e2e22d9" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250809 02:05:44.251646 28342 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.301s user 0.511s sys 0.788s
W20250809 02:05:44.251933 28342 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.302s user 0.511s sys 0.788s
W20250809 02:05:45.657469 28369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:45.658092 28369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:45.688992 28369 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250809 02:05:46.969108 28369 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.241s user 0.472s sys 0.766s
W20250809 02:05:46.969489 28369 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.241s user 0.472s sys 0.766s
W20250809 02:05:48.354451 28386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:48.355115 28386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:48.386116 28386 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250809 02:05:49.690234 28386 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.264s user 0.442s sys 0.817s
W20250809 02:05:49.690675 28386 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.265s user 0.442s sys 0.817s
W20250809 02:05:51.067219 28400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:51.067757 28400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:51.099164 28400 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250809 02:05:52.363358 28400 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.227s user 0.456s sys 0.770s
W20250809 02:05:52.363735 28400 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.227s user 0.456s sys 0.770s
I20250809 02:05:53.434199 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 27906
I20250809 02:05:53.461879 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 28039
I20250809 02:05:53.490638 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 28173
I20250809 02:05:53.519414 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 27814
2025-08-09T02:05:53Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (20046 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250809 02:05:53.579136 26343 test_util.cc:276] Using random seed: 559765605
I20250809 02:05:53.583652 26343 ts_itest-base.cc:115] Starting cluster with:
I20250809 02:05:53.583842 26343 ts_itest-base.cc:116] --------------
I20250809 02:05:53.584007 26343 ts_itest-base.cc:117] 3 tablet servers
I20250809 02:05:53.584146 26343 ts_itest-base.cc:118] 3 replicas per TS
I20250809 02:05:53.584280 26343 ts_itest-base.cc:119] --------------
2025-08-09T02:05:53Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T02:05:53Z Disabled control of system clock
I20250809 02:05:53.621729 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:38825
--webserver_interface=127.25.185.254
--webserver_port=0
--builtin_ntp_servers=127.25.185.212:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:38825
--superuser_acl=no-such-user with env {}
W20250809 02:05:53.920007 28422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:53.920629 28422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:53.921136 28422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:53.956153 28422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:05:53.956501 28422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:53.956751 28422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:05:53.956983 28422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:05:53.993575 28422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:38825
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:38825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:53.994931 28422 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:53.996678 28422 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:54.008117 28428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:55.411345 28427 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ec0 after lost signal to thread 28422
W20250809 02:05:55.801287 28422 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.793s user 0.639s sys 1.154s
W20250809 02:05:55.801679 28422 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.794s user 0.639s sys 1.154s
W20250809 02:05:54.008713 28429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:55.803660 28431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:55.806111 28430 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1793 milliseconds
I20250809 02:05:55.806147 28422 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:55.807430 28422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:55.809774 28422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:55.811141 28422 hybrid_clock.cc:648] HybridClock initialized: now 1754705155811096 us; error 42 us; skew 500 ppm
I20250809 02:05:55.811929 28422 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:55.818074 28422 webserver.cc:489] Webserver started at http://127.25.185.254:35065/ using document root <none> and password file <none>
I20250809 02:05:55.819013 28422 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:55.819229 28422 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:55.819656 28422 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:55.823796 28422 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0"
format_stamp: "Formatted at 2025-08-09 02:05:55 on dist-test-slave-24ff"
I20250809 02:05:55.824841 28422 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0"
format_stamp: "Formatted at 2025-08-09 02:05:55 on dist-test-slave-24ff"
I20250809 02:05:55.831833 28422 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250809 02:05:55.837105 28438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:55.838128 28422 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250809 02:05:55.838418 28422 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0"
format_stamp: "Formatted at 2025-08-09 02:05:55 on dist-test-slave-24ff"
I20250809 02:05:55.838781 28422 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:55.895490 28422 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:55.896975 28422 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:55.897394 28422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:55.968608 28422 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:38825
I20250809 02:05:55.968677 28489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:38825 every 8 connection(s)
I20250809 02:05:55.971235 28422 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 02:05:55.973635 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 28422
I20250809 02:05:55.974135 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 02:05:55.977356 28490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:05:55.999976 28490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0: Bootstrap starting.
I20250809 02:05:56.005946 28490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0: Neither blocks nor log segments found. Creating new log.
I20250809 02:05:56.007869 28490 log.cc:826] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0: Log is configured to *not* fsync() on all Append() calls
I20250809 02:05:56.012074 28490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0: No bootstrap required, opened a new log
I20250809 02:05:56.029973 28490 raft_consensus.cc:357] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38825 } }
I20250809 02:05:56.030630 28490 raft_consensus.cc:383] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:05:56.030905 28490 raft_consensus.cc:738] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbeb0f2a9ebe4f879d557de3e8eb13c0, State: Initialized, Role: FOLLOWER
I20250809 02:05:56.031610 28490 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38825 } }
I20250809 02:05:56.032111 28490 raft_consensus.cc:397] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:05:56.032361 28490 raft_consensus.cc:491] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:05:56.032677 28490 raft_consensus.cc:3058] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:05:56.036912 28490 raft_consensus.cc:513] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38825 } }
I20250809 02:05:56.037575 28490 leader_election.cc:304] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbeb0f2a9ebe4f879d557de3e8eb13c0; no voters:
I20250809 02:05:56.039361 28490 leader_election.cc:290] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:05:56.040108 28495 raft_consensus.cc:2802] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:05:56.042115 28495 raft_consensus.cc:695] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [term 1 LEADER]: Becoming Leader. State: Replica: dbeb0f2a9ebe4f879d557de3e8eb13c0, State: Running, Role: LEADER
I20250809 02:05:56.043466 28495 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38825 } }
I20250809 02:05:56.044603 28490 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:05:56.052634 28497 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38825 } } }
I20250809 02:05:56.053503 28497 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [sys.catalog]: This master's current role is: LEADER
I20250809 02:05:56.056422 28502 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:05:56.057092 28496 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbeb0f2a9ebe4f879d557de3e8eb13c0. Latest consensus state: current_term: 1 leader_uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbeb0f2a9ebe4f879d557de3e8eb13c0" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 38825 } } }
I20250809 02:05:56.057803 28496 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0 [sys.catalog]: This master's current role is: LEADER
I20250809 02:05:56.067360 28502 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:05:56.084551 28502 catalog_manager.cc:1349] Generated new cluster ID: 16a1fca727934991be0869da6afaf595
I20250809 02:05:56.084889 28502 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:05:56.104741 28502 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 02:05:56.106106 28502 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:05:56.127969 28502 catalog_manager.cc:5955] T 00000000000000000000000000000000 P dbeb0f2a9ebe4f879d557de3e8eb13c0: Generated new TSK 0
I20250809 02:05:56.129109 28502 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 02:05:56.149614 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:0
--local_ip_for_outbound_sockets=127.25.185.193
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38825
--builtin_ntp_servers=127.25.185.212:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250809 02:05:56.464532 28514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:56.465075 28514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:56.465561 28514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:56.497255 28514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:56.498085 28514 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:05:56.532857 28514 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:56.534229 28514 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:56.535818 28514 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:56.548132 28520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:56.553400 28523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:56.549764 28521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:58.039500 28522 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1489 milliseconds
I20250809 02:05:58.039634 28514 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:58.040838 28514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:58.043149 28514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:58.044559 28514 hybrid_clock.cc:648] HybridClock initialized: now 1754705158044531 us; error 42 us; skew 500 ppm
I20250809 02:05:58.045372 28514 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:58.052040 28514 webserver.cc:489] Webserver started at http://127.25.185.193:32891/ using document root <none> and password file <none>
I20250809 02:05:58.052963 28514 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:58.053159 28514 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:58.053647 28514 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:58.057997 28514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "962b7807b3904b0abafc61091294c2cd"
format_stamp: "Formatted at 2025-08-09 02:05:58 on dist-test-slave-24ff"
I20250809 02:05:58.059159 28514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "962b7807b3904b0abafc61091294c2cd"
format_stamp: "Formatted at 2025-08-09 02:05:58 on dist-test-slave-24ff"
I20250809 02:05:58.066790 28514 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250809 02:05:58.072809 28531 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:58.074025 28514 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20250809 02:05:58.074354 28514 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "962b7807b3904b0abafc61091294c2cd"
format_stamp: "Formatted at 2025-08-09 02:05:58 on dist-test-slave-24ff"
I20250809 02:05:58.074672 28514 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:05:58.129416 28514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:05:58.130897 28514 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:05:58.131317 28514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:05:58.134275 28514 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:05:58.138365 28514 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:05:58.138553 28514 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:58.138844 28514 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:05:58.138994 28514 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:58.304108 28514 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:44209
I20250809 02:05:58.304278 28643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:44209 every 8 connection(s)
I20250809 02:05:58.306869 28514 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 02:05:58.309839 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 28514
I20250809 02:05:58.310765 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 02:05:58.320399 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.194:0
--local_ip_for_outbound_sockets=127.25.185.194
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38825
--builtin_ntp_servers=127.25.185.212:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:05:58.338958 28644 heartbeater.cc:344] Connected to a master server at 127.25.185.254:38825
I20250809 02:05:58.339507 28644 heartbeater.cc:461] Registering TS with master...
I20250809 02:05:58.340746 28644 heartbeater.cc:507] Master 127.25.185.254:38825 requested a full tablet report, sending...
I20250809 02:05:58.343981 28455 ts_manager.cc:194] Registered new tserver with Master: 962b7807b3904b0abafc61091294c2cd (127.25.185.193:44209)
I20250809 02:05:58.346925 28455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:59103
W20250809 02:05:58.622411 28648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:05:58.622938 28648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:05:58.623435 28648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:05:58.654366 28648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:05:58.655289 28648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.194
I20250809 02:05:58.690306 28648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.194
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:05:58.691664 28648 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:05:58.693259 28648 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:05:58.704734 28654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:05:59.351347 28644 heartbeater.cc:499] Master 127.25.185.254:38825 was elected leader, sending a full tablet report...
W20250809 02:05:58.705435 28655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:59.906982 28657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:05:59.910143 28656 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1199 milliseconds
W20250809 02:05:59.910414 28648 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.205s user 0.390s sys 0.810s
W20250809 02:05:59.910884 28648 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.206s user 0.390s sys 0.814s
I20250809 02:05:59.911193 28648 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:05:59.912732 28648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:05:59.915572 28648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:05:59.917101 28648 hybrid_clock.cc:648] HybridClock initialized: now 1754705159917056 us; error 30 us; skew 500 ppm
I20250809 02:05:59.918344 28648 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:05:59.927419 28648 webserver.cc:489] Webserver started at http://127.25.185.194:35093/ using document root <none> and password file <none>
I20250809 02:05:59.928890 28648 fs_manager.cc:362] Metadata directory not provided
I20250809 02:05:59.929221 28648 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:05:59.929896 28648 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:05:59.937357 28648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "707e7b617ac9415a9191b6dfb9d68af2"
format_stamp: "Formatted at 2025-08-09 02:05:59 on dist-test-slave-24ff"
I20250809 02:05:59.939162 28648 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "707e7b617ac9415a9191b6dfb9d68af2"
format_stamp: "Formatted at 2025-08-09 02:05:59 on dist-test-slave-24ff"
I20250809 02:05:59.950224 28648 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.001s sys 0.009s
I20250809 02:05:59.958395 28664 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:05:59.959628 28648 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250809 02:05:59.960058 28648 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "707e7b617ac9415a9191b6dfb9d68af2"
format_stamp: "Formatted at 2025-08-09 02:05:59 on dist-test-slave-24ff"
I20250809 02:05:59.960528 28648 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:00.048700 28648 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:00.051043 28648 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:00.051653 28648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:00.055280 28648 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:00.059455 28648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:06:00.059660 28648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:00.059871 28648 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:06:00.060005 28648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:00.191815 28648 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.194:41979
I20250809 02:06:00.192025 28776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.194:41979 every 8 connection(s)
I20250809 02:06:00.194315 28648 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 02:06:00.201948 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 28648
I20250809 02:06:00.202363 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 02:06:00.208439 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.195:0
--local_ip_for_outbound_sockets=127.25.185.195
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38825
--builtin_ntp_servers=127.25.185.212:44257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:06:00.216289 28777 heartbeater.cc:344] Connected to a master server at 127.25.185.254:38825
I20250809 02:06:00.216738 28777 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:00.217744 28777 heartbeater.cc:507] Master 127.25.185.254:38825 requested a full tablet report, sending...
I20250809 02:06:00.219825 28455 ts_manager.cc:194] Registered new tserver with Master: 707e7b617ac9415a9191b6dfb9d68af2 (127.25.185.194:41979)
I20250809 02:06:00.221028 28455 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.194:56473
W20250809 02:06:00.601264 28781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:00.601770 28781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:00.602283 28781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:00.633409 28781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:00.634264 28781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.195
I20250809 02:06:00.668953 28781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:44257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:38825
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.195
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:00.670300 28781 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:00.671866 28781 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:00.683916 28787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:06:01.224213 28777 heartbeater.cc:499] Master 127.25.185.254:38825 was elected leader, sending a full tablet report...
W20250809 02:06:01.568815 28640 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac8a0 after lost signal to thread 28515
W20250809 02:06:01.570549 28640 debug-util.cc:398] Leaking SignalData structure 0x7b08000acd40 after lost signal to thread 28643
W20250809 02:06:00.684377 28788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:02.062086 28790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:02.065094 28789 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1377 milliseconds
I20250809 02:06:02.065219 28781 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:02.066367 28781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:02.069001 28781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:02.070410 28781 hybrid_clock.cc:648] HybridClock initialized: now 1754705162070353 us; error 67 us; skew 500 ppm
I20250809 02:06:02.071216 28781 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:02.078186 28781 webserver.cc:489] Webserver started at http://127.25.185.195:33565/ using document root <none> and password file <none>
I20250809 02:06:02.079115 28781 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:02.079315 28781 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:02.079780 28781 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:06:02.084276 28781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "efee6ce9c2c745c2aced8474a0680ca2"
format_stamp: "Formatted at 2025-08-09 02:06:02 on dist-test-slave-24ff"
I20250809 02:06:02.085345 28781 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "efee6ce9c2c745c2aced8474a0680ca2"
format_stamp: "Formatted at 2025-08-09 02:06:02 on dist-test-slave-24ff"
I20250809 02:06:02.092543 28781 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250809 02:06:02.098395 28797 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:02.099478 28781 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250809 02:06:02.099761 28781 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "efee6ce9c2c745c2aced8474a0680ca2"
format_stamp: "Formatted at 2025-08-09 02:06:02 on dist-test-slave-24ff"
I20250809 02:06:02.100077 28781 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:02.175643 28781 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:02.177099 28781 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:02.177518 28781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:02.180572 28781 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:02.184829 28781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:06:02.185039 28781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:02.185310 28781 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:06:02.185477 28781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:02.360659 28781 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.195:40609
I20250809 02:06:02.360845 28909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.195:40609 every 8 connection(s)
I20250809 02:06:02.363241 28781 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 02:06:02.364781 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 28781
I20250809 02:06:02.365366 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 02:06:02.401104 28910 heartbeater.cc:344] Connected to a master server at 127.25.185.254:38825
I20250809 02:06:02.401604 28910 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:02.402765 28910 heartbeater.cc:507] Master 127.25.185.254:38825 requested a full tablet report, sending...
I20250809 02:06:02.406030 28454 ts_manager.cc:194] Registered new tserver with Master: efee6ce9c2c745c2aced8474a0680ca2 (127.25.185.195:40609)
I20250809 02:06:02.407541 28454 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.195:57315
I20250809 02:06:02.420822 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:06:02.456642 28454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39566:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250809 02:06:02.475245 28454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 02:06:02.545977 28579 tablet_service.cc:1468] Processing CreateTablet for tablet c392bd1464a944d6a96a77846d8ef0a7 (DEFAULT_TABLE table=TestTable [id=1dd42bbf415145fa83a1f3dbf0443775]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:06:02.548285 28579 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c392bd1464a944d6a96a77846d8ef0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:02.552062 28712 tablet_service.cc:1468] Processing CreateTablet for tablet c392bd1464a944d6a96a77846d8ef0a7 (DEFAULT_TABLE table=TestTable [id=1dd42bbf415145fa83a1f3dbf0443775]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:06:02.554155 28712 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c392bd1464a944d6a96a77846d8ef0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:02.557633 28845 tablet_service.cc:1468] Processing CreateTablet for tablet c392bd1464a944d6a96a77846d8ef0a7 (DEFAULT_TABLE table=TestTable [id=1dd42bbf415145fa83a1f3dbf0443775]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:06:02.559525 28845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c392bd1464a944d6a96a77846d8ef0a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:02.588116 28929 tablet_bootstrap.cc:492] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2: Bootstrap starting.
I20250809 02:06:02.598304 28931 tablet_bootstrap.cc:492] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2: Bootstrap starting.
I20250809 02:06:02.599242 28929 tablet_bootstrap.cc:654] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2: Neither blocks nor log segments found. Creating new log.
I20250809 02:06:02.600157 28930 tablet_bootstrap.cc:492] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd: Bootstrap starting.
I20250809 02:06:02.602648 28929 log.cc:826] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:02.607059 28931 tablet_bootstrap.cc:654] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2: Neither blocks nor log segments found. Creating new log.
I20250809 02:06:02.608201 28930 tablet_bootstrap.cc:654] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd: Neither blocks nor log segments found. Creating new log.
I20250809 02:06:02.610512 28931 log.cc:826] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:02.610790 28930 log.cc:826] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:02.614944 28929 tablet_bootstrap.cc:492] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2: No bootstrap required, opened a new log
I20250809 02:06:02.615630 28929 ts_tablet_manager.cc:1397] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2: Time spent bootstrapping tablet: real 0.028s user 0.022s sys 0.000s
I20250809 02:06:02.621518 28931 tablet_bootstrap.cc:492] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2: No bootstrap required, opened a new log
I20250809 02:06:02.622040 28931 ts_tablet_manager.cc:1397] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.004s
I20250809 02:06:02.638187 28930 tablet_bootstrap.cc:492] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd: No bootstrap required, opened a new log
I20250809 02:06:02.638808 28930 ts_tablet_manager.cc:1397] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd: Time spent bootstrapping tablet: real 0.039s user 0.016s sys 0.022s
I20250809 02:06:02.647514 28931 raft_consensus.cc:357] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707e7b617ac9415a9191b6dfb9d68af2" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41979 } } peers { permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 } } peers { permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 } }
I20250809 02:06:02.647509 28929 raft_consensus.cc:357] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707e7b617ac9415a9191b6dfb9d68af2" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41979 } } peers { permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 } } peers { permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 } }
I20250809 02:06:02.648452 28929 raft_consensus.cc:383] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:06:02.648501 28931 raft_consensus.cc:383] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:06:02.648779 28931 raft_consensus.cc:738] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efee6ce9c2c745c2aced8474a0680ca2, State: Initialized, Role: FOLLOWER
I20250809 02:06:02.648782 28929 raft_consensus.cc:738] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 707e7b617ac9415a9191b6dfb9d68af2, State: Initialized, Role: FOLLOWER
I20250809 02:06:02.649730 28931 consensus_queue.cc:260] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707e7b617ac9415a9191b6dfb9d68af2" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41979 } } peers { permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 } } peers { permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 } }
I20250809 02:06:02.649959 28929 consensus_queue.cc:260] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707e7b617ac9415a9191b6dfb9d68af2" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41979 } } peers { permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 } } peers { permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 } }
I20250809 02:06:02.657860 28929 ts_tablet_manager.cc:1428] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2: Time spent starting tablet: real 0.042s user 0.035s sys 0.000s
I20250809 02:06:02.667801 28930 raft_consensus.cc:357] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707e7b617ac9415a9191b6dfb9d68af2" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41979 } } peers { permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 } } peers { permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 } }
I20250809 02:06:02.668910 28930 raft_consensus.cc:383] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:06:02.669296 28930 raft_consensus.cc:738] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 962b7807b3904b0abafc61091294c2cd, State: Initialized, Role: FOLLOWER
I20250809 02:06:02.670390 28930 consensus_queue.cc:260] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707e7b617ac9415a9191b6dfb9d68af2" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41979 } } peers { permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 } } peers { permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 } }
I20250809 02:06:02.678337 28910 heartbeater.cc:499] Master 127.25.185.254:38825 was elected leader, sending a full tablet report...
I20250809 02:06:02.680799 28930 ts_tablet_manager.cc:1428] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd: Time spent starting tablet: real 0.042s user 0.030s sys 0.008s
I20250809 02:06:02.681919 28931 ts_tablet_manager.cc:1428] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2: Time spent starting tablet: real 0.060s user 0.034s sys 0.025s
W20250809 02:06:02.701366 28778 tablet.cc:2378] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 02:06:02.823487 28645 tablet.cc:2378] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:06:02.857525 28936 raft_consensus.cc:491] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:06:02.858178 28936 raft_consensus.cc:513] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707e7b617ac9415a9191b6dfb9d68af2" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41979 } } peers { permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 } } peers { permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 } }
I20250809 02:06:02.861526 28936 leader_election.cc:290] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 962b7807b3904b0abafc61091294c2cd (127.25.185.193:44209), efee6ce9c2c745c2aced8474a0680ca2 (127.25.185.195:40609)
W20250809 02:06:02.878654 28911 tablet.cc:2378] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:06:02.878523 28599 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c392bd1464a944d6a96a77846d8ef0a7" candidate_uuid: "707e7b617ac9415a9191b6dfb9d68af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962b7807b3904b0abafc61091294c2cd" is_pre_election: true
I20250809 02:06:02.878533 28865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c392bd1464a944d6a96a77846d8ef0a7" candidate_uuid: "707e7b617ac9415a9191b6dfb9d68af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efee6ce9c2c745c2aced8474a0680ca2" is_pre_election: true
I20250809 02:06:02.879455 28865 raft_consensus.cc:2466] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 707e7b617ac9415a9191b6dfb9d68af2 in term 0.
I20250809 02:06:02.879458 28599 raft_consensus.cc:2466] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 707e7b617ac9415a9191b6dfb9d68af2 in term 0.
I20250809 02:06:02.880959 28668 leader_election.cc:304] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 707e7b617ac9415a9191b6dfb9d68af2, efee6ce9c2c745c2aced8474a0680ca2; no voters:
I20250809 02:06:02.881692 28936 raft_consensus.cc:2802] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 02:06:02.882020 28936 raft_consensus.cc:491] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 02:06:02.882292 28936 raft_consensus.cc:3058] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:06:02.886515 28936 raft_consensus.cc:513] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707e7b617ac9415a9191b6dfb9d68af2" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41979 } } peers { permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 } } peers { permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 } }
I20250809 02:06:02.887914 28936 leader_election.cc:290] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [CANDIDATE]: Term 1 election: Requested vote from peers 962b7807b3904b0abafc61091294c2cd (127.25.185.193:44209), efee6ce9c2c745c2aced8474a0680ca2 (127.25.185.195:40609)
I20250809 02:06:02.888844 28599 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c392bd1464a944d6a96a77846d8ef0a7" candidate_uuid: "707e7b617ac9415a9191b6dfb9d68af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "962b7807b3904b0abafc61091294c2cd"
I20250809 02:06:02.889022 28865 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c392bd1464a944d6a96a77846d8ef0a7" candidate_uuid: "707e7b617ac9415a9191b6dfb9d68af2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efee6ce9c2c745c2aced8474a0680ca2"
I20250809 02:06:02.889281 28599 raft_consensus.cc:3058] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:06:02.889586 28865 raft_consensus.cc:3058] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:06:02.895850 28599 raft_consensus.cc:2466] T c392bd1464a944d6a96a77846d8ef0a7 P 962b7807b3904b0abafc61091294c2cd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 707e7b617ac9415a9191b6dfb9d68af2 in term 1.
I20250809 02:06:02.896889 28667 leader_election.cc:304] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 707e7b617ac9415a9191b6dfb9d68af2, 962b7807b3904b0abafc61091294c2cd; no voters:
I20250809 02:06:02.897220 28865 raft_consensus.cc:2466] T c392bd1464a944d6a96a77846d8ef0a7 P efee6ce9c2c745c2aced8474a0680ca2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 707e7b617ac9415a9191b6dfb9d68af2 in term 1.
I20250809 02:06:02.897533 28936 raft_consensus.cc:2802] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:06:02.899739 28936 raft_consensus.cc:695] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [term 1 LEADER]: Becoming Leader. State: Replica: 707e7b617ac9415a9191b6dfb9d68af2, State: Running, Role: LEADER
I20250809 02:06:02.900962 28936 consensus_queue.cc:237] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707e7b617ac9415a9191b6dfb9d68af2" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41979 } } peers { permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 } } peers { permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 } }
I20250809 02:06:02.911980 28454 catalog_manager.cc:5582] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 707e7b617ac9415a9191b6dfb9d68af2 (127.25.185.194). New cstate: current_term: 1 leader_uuid: "707e7b617ac9415a9191b6dfb9d68af2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "707e7b617ac9415a9191b6dfb9d68af2" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 41979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 } health_report { overall_health: UNKNOWN } } }
I20250809 02:06:02.993907 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:06:02.998332 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 962b7807b3904b0abafc61091294c2cd to finish bootstrapping
I20250809 02:06:03.014357 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 707e7b617ac9415a9191b6dfb9d68af2 to finish bootstrapping
I20250809 02:06:03.027309 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver efee6ce9c2c745c2aced8474a0680ca2 to finish bootstrapping
I20250809 02:06:03.410027 28936 consensus_queue.cc:1035] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efee6ce9c2c745c2aced8474a0680ca2" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 40609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250809 02:06:03.455984 28950 consensus_queue.cc:1035] T c392bd1464a944d6a96a77846d8ef0a7 P 707e7b617ac9415a9191b6dfb9d68af2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "962b7807b3904b0abafc61091294c2cd" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 44209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.007s
W20250809 02:06:04.724347 28454 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:39582
I20250809 02:06:05.754815 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 28514
I20250809 02:06:05.781699 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 28648
I20250809 02:06:05.809626 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 28781
I20250809 02:06:05.835366 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 28422
2025-08-09T02:06:05Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (12307 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250809 02:06:05.887090 26343 test_util.cc:276] Using random seed: 572073560
I20250809 02:06:05.891010 26343 ts_itest-base.cc:115] Starting cluster with:
I20250809 02:06:05.891170 26343 ts_itest-base.cc:116] --------------
I20250809 02:06:05.891338 26343 ts_itest-base.cc:117] 3 tablet servers
I20250809 02:06:05.891489 26343 ts_itest-base.cc:118] 3 replicas per TS
I20250809 02:06:05.891630 26343 ts_itest-base.cc:119] --------------
2025-08-09T02:06:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T02:06:05Z Disabled control of system clock
I20250809 02:06:05.928023 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:43523
--webserver_interface=127.25.185.254
--webserver_port=0
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:43523 with env {}
W20250809 02:06:06.225106 28979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:06.225646 28979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:06.226020 28979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:06.256798 28979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:06:06.257125 28979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:06.257330 28979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:06:06.257526 28979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:06:06.292497 28979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:43523
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:06.293823 28979 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:06.295454 28979 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:06.305387 28985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:06.307065 28986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:07.517318 28988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:07.520314 28987 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1208 milliseconds
I20250809 02:06:07.520407 28979 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:07.521598 28979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:07.524564 28979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:07.525962 28979 hybrid_clock.cc:648] HybridClock initialized: now 1754705167525904 us; error 59 us; skew 500 ppm
I20250809 02:06:07.526772 28979 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:07.533154 28979 webserver.cc:489] Webserver started at http://127.25.185.254:43033/ using document root <none> and password file <none>
I20250809 02:06:07.534018 28979 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:07.534225 28979 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:07.534626 28979 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:06:07.538895 28979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "bde1b90c638440a6acb697035e19e66e"
format_stamp: "Formatted at 2025-08-09 02:06:07 on dist-test-slave-24ff"
I20250809 02:06:07.539933 28979 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "bde1b90c638440a6acb697035e19e66e"
format_stamp: "Formatted at 2025-08-09 02:06:07 on dist-test-slave-24ff"
I20250809 02:06:07.546746 28979 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250809 02:06:07.552150 28995 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:07.553189 28979 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250809 02:06:07.553496 28979 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bde1b90c638440a6acb697035e19e66e"
format_stamp: "Formatted at 2025-08-09 02:06:07 on dist-test-slave-24ff"
I20250809 02:06:07.553797 28979 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:07.629094 28979 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:07.630582 28979 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:07.630978 28979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:07.699693 28979 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:43523
I20250809 02:06:07.699772 29046 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:43523 every 8 connection(s)
I20250809 02:06:07.702291 28979 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 02:06:07.707026 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 28979
I20250809 02:06:07.707530 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 02:06:07.708245 29047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:07.733462 29047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Bootstrap starting.
I20250809 02:06:07.739480 29047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Neither blocks nor log segments found. Creating new log.
I20250809 02:06:07.741137 29047 log.cc:826] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:07.745394 29047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: No bootstrap required, opened a new log
I20250809 02:06:07.762292 29047 raft_consensus.cc:357] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:07.763000 29047 raft_consensus.cc:383] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:06:07.763203 29047 raft_consensus.cc:738] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde1b90c638440a6acb697035e19e66e, State: Initialized, Role: FOLLOWER
I20250809 02:06:07.763805 29047 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:07.764279 29047 raft_consensus.cc:397] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:07.764539 29047 raft_consensus.cc:491] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:07.764799 29047 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:06:07.768764 29047 raft_consensus.cc:513] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:07.769424 29047 leader_election.cc:304] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bde1b90c638440a6acb697035e19e66e; no voters:
I20250809 02:06:07.771342 29047 leader_election.cc:290] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:06:07.772068 29052 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:06:07.774060 29052 raft_consensus.cc:695] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 1 LEADER]: Becoming Leader. State: Replica: bde1b90c638440a6acb697035e19e66e, State: Running, Role: LEADER
I20250809 02:06:07.774837 29052 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:07.775475 29047 sys_catalog.cc:564] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:06:07.785509 29053 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bde1b90c638440a6acb697035e19e66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } } }
I20250809 02:06:07.786154 29053 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: This master's current role is: LEADER
I20250809 02:06:07.788480 29054 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde1b90c638440a6acb697035e19e66e. Latest consensus state: current_term: 1 leader_uuid: "bde1b90c638440a6acb697035e19e66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } } }
I20250809 02:06:07.789146 29054 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: This master's current role is: LEADER
I20250809 02:06:07.793047 29062 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:06:07.802999 29062 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:06:07.820101 29062 catalog_manager.cc:1349] Generated new cluster ID: d7cf4e86ee38451fa2ceee0cdb5e54a4
I20250809 02:06:07.820359 29062 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:06:07.843254 29062 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 02:06:07.845221 29062 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:06:07.871829 29062 catalog_manager.cc:5955] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Generated new TSK 0
I20250809 02:06:07.872761 29062 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 02:06:07.886106 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:0
--local_ip_for_outbound_sockets=127.25.185.193
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:43523
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250809 02:06:08.188783 29071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:08.189320 29071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:08.189834 29071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:08.221706 29071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:08.222563 29071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:06:08.257723 29071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:43523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:08.259075 29071 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:08.260753 29071 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:08.273536 29078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:08.279196 29081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:08.276595 29079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:09.480161 29080 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1200 milliseconds
I20250809 02:06:09.480259 29071 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:09.481364 29071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:09.483588 29071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:09.485061 29071 hybrid_clock.cc:648] HybridClock initialized: now 1754705169485026 us; error 73 us; skew 500 ppm
I20250809 02:06:09.485862 29071 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:09.492852 29071 webserver.cc:489] Webserver started at http://127.25.185.193:43221/ using document root <none> and password file <none>
I20250809 02:06:09.493764 29071 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:09.493975 29071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:09.494411 29071 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:06:09.498792 29071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d8d01767213e461c8b872b0b1e439fc1"
format_stamp: "Formatted at 2025-08-09 02:06:09 on dist-test-slave-24ff"
I20250809 02:06:09.499869 29071 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d8d01767213e461c8b872b0b1e439fc1"
format_stamp: "Formatted at 2025-08-09 02:06:09 on dist-test-slave-24ff"
I20250809 02:06:09.507380 29071 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250809 02:06:09.513336 29088 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:09.514457 29071 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.002s
I20250809 02:06:09.514833 29071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d8d01767213e461c8b872b0b1e439fc1"
format_stamp: "Formatted at 2025-08-09 02:06:09 on dist-test-slave-24ff"
I20250809 02:06:09.515133 29071 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:09.565073 29071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:09.566517 29071 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:09.566962 29071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:09.570024 29071 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:09.574739 29071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:06:09.575026 29071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250809 02:06:09.575291 29071 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:06:09.575448 29071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:09.745393 29071 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:41085
I20250809 02:06:09.745491 29200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:41085 every 8 connection(s)
I20250809 02:06:09.748136 29071 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 02:06:09.752815 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 29071
I20250809 02:06:09.753417 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 02:06:09.763546 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.194:0
--local_ip_for_outbound_sockets=127.25.185.194
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:43523
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:06:09.790234 29201 heartbeater.cc:344] Connected to a master server at 127.25.185.254:43523
I20250809 02:06:09.790829 29201 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:09.792207 29201 heartbeater.cc:507] Master 127.25.185.254:43523 requested a full tablet report, sending...
I20250809 02:06:09.795595 29012 ts_manager.cc:194] Registered new tserver with Master: d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193:41085)
I20250809 02:06:09.798364 29012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:34649
W20250809 02:06:10.075279 29205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:10.075802 29205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:10.076318 29205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:10.107601 29205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:10.108449 29205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.194
I20250809 02:06:10.143128 29205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:43523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.194
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:10.144439 29205 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:10.146039 29205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:10.158298 29211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:06:10.802846 29201 heartbeater.cc:499] Master 127.25.185.254:43523 was elected leader, sending a full tablet report...
W20250809 02:06:10.160184 29212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:10.163321 29214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:11.349093 29213 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1185 milliseconds
I20250809 02:06:11.349210 29205 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:11.350507 29205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:11.353221 29205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:11.354712 29205 hybrid_clock.cc:648] HybridClock initialized: now 1754705171354640 us; error 58 us; skew 500 ppm
I20250809 02:06:11.355491 29205 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:11.362099 29205 webserver.cc:489] Webserver started at http://127.25.185.194:41475/ using document root <none> and password file <none>
I20250809 02:06:11.363065 29205 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:11.363266 29205 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:11.363776 29205 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:06:11.368302 29205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "14f06d05002142c48d6482f6b3cd0791"
format_stamp: "Formatted at 2025-08-09 02:06:11 on dist-test-slave-24ff"
I20250809 02:06:11.369367 29205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "14f06d05002142c48d6482f6b3cd0791"
format_stamp: "Formatted at 2025-08-09 02:06:11 on dist-test-slave-24ff"
I20250809 02:06:11.376334 29205 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250809 02:06:11.381916 29221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:11.383013 29205 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250809 02:06:11.383361 29205 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "14f06d05002142c48d6482f6b3cd0791"
format_stamp: "Formatted at 2025-08-09 02:06:11 on dist-test-slave-24ff"
I20250809 02:06:11.383796 29205 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:11.430261 29205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:11.431845 29205 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:11.432284 29205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:11.434731 29205 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:11.438860 29205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:06:11.439079 29205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:11.439334 29205 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:06:11.439495 29205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:11.573844 29205 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.194:38491
I20250809 02:06:11.573942 29333 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.194:38491 every 8 connection(s)
I20250809 02:06:11.576404 29205 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 02:06:11.586653 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 29205
I20250809 02:06:11.587193 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 02:06:11.600350 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.195:0
--local_ip_for_outbound_sockets=127.25.185.195
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:43523
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:06:11.601447 29334 heartbeater.cc:344] Connected to a master server at 127.25.185.254:43523
I20250809 02:06:11.601881 29334 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:11.602983 29334 heartbeater.cc:507] Master 127.25.185.254:43523 requested a full tablet report, sending...
I20250809 02:06:11.605186 29012 ts_manager.cc:194] Registered new tserver with Master: 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491)
I20250809 02:06:11.606424 29012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.194:40447
W20250809 02:06:11.909886 29338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:11.910352 29338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:11.910826 29338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:11.942518 29338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:11.943370 29338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.195
I20250809 02:06:11.982429 29338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:43523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.195
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:11.983790 29338 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:11.985412 29338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:11.997442 29344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:06:12.609771 29334 heartbeater.cc:499] Master 127.25.185.254:43523 was elected leader, sending a full tablet report...
W20250809 02:06:11.999028 29345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:13.402307 29343 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 29338
W20250809 02:06:13.709221 29338 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.711s user 0.647s sys 1.034s
W20250809 02:06:13.711325 29338 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.713s user 0.647s sys 1.034s
W20250809 02:06:13.712522 29347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:13.714414 29346 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1711 milliseconds
I20250809 02:06:13.714447 29338 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:13.715875 29338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:13.718451 29338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:13.720009 29338 hybrid_clock.cc:648] HybridClock initialized: now 1754705173719985 us; error 45 us; skew 500 ppm
I20250809 02:06:13.721179 29338 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:13.728268 29338 webserver.cc:489] Webserver started at http://127.25.185.195:42581/ using document root <none> and password file <none>
I20250809 02:06:13.729434 29338 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:13.729702 29338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:13.730206 29338 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:06:13.736301 29338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3ffc0ee6c47f460cbfa594f7a260c685"
format_stamp: "Formatted at 2025-08-09 02:06:13 on dist-test-slave-24ff"
I20250809 02:06:13.737709 29338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3ffc0ee6c47f460cbfa594f7a260c685"
format_stamp: "Formatted at 2025-08-09 02:06:13 on dist-test-slave-24ff"
I20250809 02:06:13.745711 29338 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.010s sys 0.000s
I20250809 02:06:13.752125 29354 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:13.753216 29338 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250809 02:06:13.753517 29338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3ffc0ee6c47f460cbfa594f7a260c685"
format_stamp: "Formatted at 2025-08-09 02:06:13 on dist-test-slave-24ff"
I20250809 02:06:13.753829 29338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:13.811204 29338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:13.812631 29338 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:13.813056 29338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:13.815539 29338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:13.820040 29338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:06:13.820303 29338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:13.820631 29338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:06:13.820797 29338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:13.954326 29338 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.195:39413
I20250809 02:06:13.954460 29466 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.195:39413 every 8 connection(s)
I20250809 02:06:13.956830 29338 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 02:06:13.961654 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 29338
I20250809 02:06:13.962164 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 02:06:13.984683 29467 heartbeater.cc:344] Connected to a master server at 127.25.185.254:43523
I20250809 02:06:13.985170 29467 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:13.986212 29467 heartbeater.cc:507] Master 127.25.185.254:43523 requested a full tablet report, sending...
I20250809 02:06:13.988747 29012 ts_manager.cc:194] Registered new tserver with Master: 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413)
I20250809 02:06:13.990644 29012 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.195:51813
I20250809 02:06:13.998991 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:06:14.038272 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:06:14.038618 26343 test_util.cc:276] Using random seed: 580225092
I20250809 02:06:14.078323 29012 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39774:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250809 02:06:14.119185 29402 tablet_service.cc:1468] Processing CreateTablet for tablet 895362632c774235a3224e8af3008462 (DEFAULT_TABLE table=TestTable [id=e0b79ecfdbd84ef8910a945ad832700b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:06:14.120643 29402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 895362632c774235a3224e8af3008462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:14.140440 29487 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap starting.
I20250809 02:06:14.145808 29487 tablet_bootstrap.cc:654] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Neither blocks nor log segments found. Creating new log.
I20250809 02:06:14.147612 29487 log.cc:826] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:14.152395 29487 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: No bootstrap required, opened a new log
I20250809 02:06:14.152814 29487 ts_tablet_manager.cc:1397] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20250809 02:06:14.169749 29487 raft_consensus.cc:357] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } }
I20250809 02:06:14.170298 29487 raft_consensus.cc:383] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:06:14.170523 29487 raft_consensus.cc:738] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffc0ee6c47f460cbfa594f7a260c685, State: Initialized, Role: FOLLOWER
I20250809 02:06:14.171182 29487 consensus_queue.cc:260] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } }
I20250809 02:06:14.171664 29487 raft_consensus.cc:397] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:14.171890 29487 raft_consensus.cc:491] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:14.172151 29487 raft_consensus.cc:3058] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:06:14.176824 29487 raft_consensus.cc:513] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } }
I20250809 02:06:14.177450 29487 leader_election.cc:304] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ffc0ee6c47f460cbfa594f7a260c685; no voters:
I20250809 02:06:14.179239 29487 leader_election.cc:290] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:06:14.179700 29489 raft_consensus.cc:2802] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:06:14.182024 29489 raft_consensus.cc:695] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 1 LEADER]: Becoming Leader. State: Replica: 3ffc0ee6c47f460cbfa594f7a260c685, State: Running, Role: LEADER
I20250809 02:06:14.182335 29467 heartbeater.cc:499] Master 127.25.185.254:43523 was elected leader, sending a full tablet report...
I20250809 02:06:14.183210 29487 ts_tablet_manager.cc:1428] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Time spent starting tablet: real 0.030s user 0.017s sys 0.015s
I20250809 02:06:14.182996 29489 consensus_queue.cc:237] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } }
I20250809 02:06:14.194993 29012 catalog_manager.cc:5582] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195). New cstate: current_term: 1 leader_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } health_report { overall_health: HEALTHY } } }
I20250809 02:06:14.394841 26343 test_util.cc:276] Using random seed: 580581313
I20250809 02:06:14.415771 29003 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39776:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250809 02:06:14.443826 29269 tablet_service.cc:1468] Processing CreateTablet for tablet 97b1f8c5f1234b6b85687619bc22c978 (DEFAULT_TABLE table=TestTable1 [id=4a9c3526c2f64a46935d37badfe61f77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:06:14.445245 29269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b1f8c5f1234b6b85687619bc22c978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:14.464156 29508 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap starting.
I20250809 02:06:14.469902 29508 tablet_bootstrap.cc:654] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Neither blocks nor log segments found. Creating new log.
I20250809 02:06:14.471580 29508 log.cc:826] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:14.475716 29508 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: No bootstrap required, opened a new log
I20250809 02:06:14.476191 29508 ts_tablet_manager.cc:1397] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Time spent bootstrapping tablet: real 0.012s user 0.011s sys 0.000s
I20250809 02:06:14.493561 29508 raft_consensus.cc:357] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } }
I20250809 02:06:14.494192 29508 raft_consensus.cc:383] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:06:14.494423 29508 raft_consensus.cc:738] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f06d05002142c48d6482f6b3cd0791, State: Initialized, Role: FOLLOWER
I20250809 02:06:14.495151 29508 consensus_queue.cc:260] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } }
I20250809 02:06:14.495685 29508 raft_consensus.cc:397] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:14.495935 29508 raft_consensus.cc:491] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:14.496235 29508 raft_consensus.cc:3058] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:06:14.500324 29508 raft_consensus.cc:513] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } }
I20250809 02:06:14.500979 29508 leader_election.cc:304] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14f06d05002142c48d6482f6b3cd0791; no voters:
I20250809 02:06:14.502812 29508 leader_election.cc:290] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:06:14.503202 29510 raft_consensus.cc:2802] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:06:14.506556 29510 raft_consensus.cc:695] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 1 LEADER]: Becoming Leader. State: Replica: 14f06d05002142c48d6482f6b3cd0791, State: Running, Role: LEADER
I20250809 02:06:14.506836 29508 ts_tablet_manager.cc:1428] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Time spent starting tablet: real 0.030s user 0.024s sys 0.008s
I20250809 02:06:14.507377 29510 consensus_queue.cc:237] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } }
I20250809 02:06:14.518409 29003 catalog_manager.cc:5582] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 reported cstate change: term changed from 0 to 1, leader changed from <none> to 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194). New cstate: current_term: 1 leader_uuid: "14f06d05002142c48d6482f6b3cd0791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } health_report { overall_health: HEALTHY } } }
I20250809 02:06:14.773535 26343 test_util.cc:276] Using random seed: 580959996
I20250809 02:06:14.800801 29005 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39782:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250809 02:06:14.829957 29136 tablet_service.cc:1468] Processing CreateTablet for tablet cc5e6748fb43459899958a80681b884a (DEFAULT_TABLE table=TestTable2 [id=d2b6cd52815d4b65af37cab8eed4ea47]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:06:14.831694 29136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cc5e6748fb43459899958a80681b884a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:14.850704 29530 tablet_bootstrap.cc:492] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Bootstrap starting.
I20250809 02:06:14.856379 29530 tablet_bootstrap.cc:654] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Neither blocks nor log segments found. Creating new log.
I20250809 02:06:14.858095 29530 log.cc:826] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:14.862629 29530 tablet_bootstrap.cc:492] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: No bootstrap required, opened a new log
I20250809 02:06:14.863155 29530 ts_tablet_manager.cc:1397] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250809 02:06:14.880067 29530 raft_consensus.cc:357] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:14.880616 29530 raft_consensus.cc:383] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:06:14.880884 29530 raft_consensus.cc:738] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Initialized, Role: FOLLOWER
I20250809 02:06:14.881574 29530 consensus_queue.cc:260] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:14.882447 29530 raft_consensus.cc:397] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:14.882738 29530 raft_consensus.cc:491] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:14.883095 29530 raft_consensus.cc:3058] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:06:14.887125 29530 raft_consensus.cc:513] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:14.887917 29530 leader_election.cc:304] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d01767213e461c8b872b0b1e439fc1; no voters:
I20250809 02:06:14.890028 29530 leader_election.cc:290] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:06:14.890368 29532 raft_consensus.cc:2802] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:06:14.892926 29532 raft_consensus.cc:695] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 1 LEADER]: Becoming Leader. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Running, Role: LEADER
I20250809 02:06:14.893708 29532 consensus_queue.cc:237] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:14.894155 29530 ts_tablet_manager.cc:1428] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Time spent starting tablet: real 0.031s user 0.019s sys 0.011s
I20250809 02:06:14.904548 29005 catalog_manager.cc:5582] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193). New cstate: current_term: 1 leader_uuid: "d8d01767213e461c8b872b0b1e439fc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } health_report { overall_health: HEALTHY } } }
I20250809 02:06:15.111195 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 28979
W20250809 02:06:15.209606 29467 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:43523 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:43523: connect: Connection refused (error 111)
W20250809 02:06:15.533746 29334 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:43523 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:43523: connect: Connection refused (error 111)
W20250809 02:06:15.922235 29201 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:43523 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:43523: connect: Connection refused (error 111)
I20250809 02:06:19.923571 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 29071
I20250809 02:06:19.948015 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 29205
I20250809 02:06:19.976660 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 29338
I20250809 02:06:20.002882 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:43523
--webserver_interface=127.25.185.254
--webserver_port=43033
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:43523 with env {}
W20250809 02:06:20.300432 29609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:20.301179 29609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:20.301715 29609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:20.332288 29609 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:06:20.332644 29609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:20.332904 29609 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:06:20.333153 29609 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:06:20.367247 29609 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:43523
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=43033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:20.368566 29609 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:20.370193 29609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:20.380470 29615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:20.381253 29616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:21.576329 29618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:21.578559 29617 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1192 milliseconds
I20250809 02:06:21.578732 29609 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:21.579988 29609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:21.582552 29609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:21.583982 29609 hybrid_clock.cc:648] HybridClock initialized: now 1754705181583941 us; error 44 us; skew 500 ppm
I20250809 02:06:21.584820 29609 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:21.591142 29609 webserver.cc:489] Webserver started at http://127.25.185.254:43033/ using document root <none> and password file <none>
I20250809 02:06:21.592088 29609 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:21.592312 29609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:21.599972 29609 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250809 02:06:21.604214 29625 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:21.605216 29609 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250809 02:06:21.605537 29609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bde1b90c638440a6acb697035e19e66e"
format_stamp: "Formatted at 2025-08-09 02:06:07 on dist-test-slave-24ff"
I20250809 02:06:21.607491 29609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:21.667946 29609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:21.669390 29609 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:21.669852 29609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:21.739107 29609 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:43523
I20250809 02:06:21.739156 29676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:43523 every 8 connection(s)
I20250809 02:06:21.742039 29609 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 02:06:21.751803 29677 sys_catalog.cc:263] Verifying existing consensus state
I20250809 02:06:21.752346 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 29609
I20250809 02:06:21.754220 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:41085
--local_ip_for_outbound_sockets=127.25.185.193
--tserver_master_addrs=127.25.185.254:43523
--webserver_port=43221
--webserver_interface=127.25.185.193
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:06:21.759688 29677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Bootstrap starting.
I20250809 02:06:21.771199 29677 log.cc:826] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:21.817494 29677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:21.818279 29677 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Bootstrap complete.
I20250809 02:06:21.838323 29677 raft_consensus.cc:357] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:21.840934 29677 raft_consensus.cc:738] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde1b90c638440a6acb697035e19e66e, State: Initialized, Role: FOLLOWER
I20250809 02:06:21.841830 29677 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:21.842331 29677 raft_consensus.cc:397] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:21.842607 29677 raft_consensus.cc:491] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:21.842962 29677 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 2 FOLLOWER]: Advancing to term 3
I20250809 02:06:21.848464 29677 raft_consensus.cc:513] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:21.849068 29677 leader_election.cc:304] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bde1b90c638440a6acb697035e19e66e; no voters:
I20250809 02:06:21.851178 29677 leader_election.cc:290] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [CANDIDATE]: Term 3 election: Requested vote from peers
I20250809 02:06:21.851554 29681 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 3 FOLLOWER]: Leader election won for term 3
I20250809 02:06:21.855017 29681 raft_consensus.cc:695] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 3 LEADER]: Becoming Leader. State: Replica: bde1b90c638440a6acb697035e19e66e, State: Running, Role: LEADER
I20250809 02:06:21.855871 29681 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:21.856240 29677 sys_catalog.cc:564] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:06:21.867045 29683 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde1b90c638440a6acb697035e19e66e. Latest consensus state: current_term: 3 leader_uuid: "bde1b90c638440a6acb697035e19e66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } } }
I20250809 02:06:21.868882 29682 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "bde1b90c638440a6acb697035e19e66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } } }
I20250809 02:06:21.869639 29682 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: This master's current role is: LEADER
I20250809 02:06:21.873418 29683 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: This master's current role is: LEADER
I20250809 02:06:21.875415 29688 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:06:21.890117 29688 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=03aaab86d51b43a3b06c26154cc9ee73]
I20250809 02:06:21.892442 29688 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=d2b6cd52815d4b65af37cab8eed4ea47]
I20250809 02:06:21.894598 29688 catalog_manager.cc:671] Loaded metadata for table TestTable [id=dfd033d43bf44fd1a68ac031ec254137]
I20250809 02:06:21.904878 29688 tablet_loader.cc:96] loaded metadata for tablet 895362632c774235a3224e8af3008462 (table TestTable [id=dfd033d43bf44fd1a68ac031ec254137])
I20250809 02:06:21.906594 29688 tablet_loader.cc:96] loaded metadata for tablet 97b1f8c5f1234b6b85687619bc22c978 (table TestTable1 [id=03aaab86d51b43a3b06c26154cc9ee73])
I20250809 02:06:21.908393 29688 tablet_loader.cc:96] loaded metadata for tablet cc5e6748fb43459899958a80681b884a (table TestTable2 [id=d2b6cd52815d4b65af37cab8eed4ea47])
I20250809 02:06:21.910408 29688 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:06:21.917191 29688 catalog_manager.cc:1261] Loaded cluster ID: d7cf4e86ee38451fa2ceee0cdb5e54a4
I20250809 02:06:21.917608 29688 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:06:21.927965 29688 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:06:21.934988 29688 catalog_manager.cc:5966] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Loaded TSK: 0
I20250809 02:06:21.937143 29688 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 02:06:21.955917 29698 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250809 02:06:22.100854 29679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:22.101354 29679 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:22.101855 29679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:22.132499 29679 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:22.133302 29679 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:06:22.167671 29679 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:41085
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=43221
--tserver_master_addrs=127.25.185.254:43523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:22.169009 29679 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:22.170655 29679 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:22.184020 29704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:22.185328 29705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:23.756737 29707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:23.759997 29706 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1572 milliseconds
I20250809 02:06:23.760063 29679 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:23.761304 29679 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:23.763523 29679 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:23.764861 29679 hybrid_clock.cc:648] HybridClock initialized: now 1754705183764832 us; error 42 us; skew 500 ppm
I20250809 02:06:23.765605 29679 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:23.771481 29679 webserver.cc:489] Webserver started at http://127.25.185.193:43221/ using document root <none> and password file <none>
I20250809 02:06:23.772387 29679 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:23.772620 29679 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:23.780042 29679 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250809 02:06:23.784458 29714 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:23.785499 29679 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250809 02:06:23.785815 29679 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d8d01767213e461c8b872b0b1e439fc1"
format_stamp: "Formatted at 2025-08-09 02:06:09 on dist-test-slave-24ff"
I20250809 02:06:23.787745 29679 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:23.840154 29679 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:23.841648 29679 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:23.842105 29679 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:23.844586 29679 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:23.850260 29721 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 02:06:23.857409 29679 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 02:06:23.857681 29679 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250809 02:06:23.857924 29679 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 02:06:23.862421 29679 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 02:06:23.862607 29679 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.001s sys 0.000s
I20250809 02:06:23.862984 29721 tablet_bootstrap.cc:492] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Bootstrap starting.
I20250809 02:06:23.913952 29721 log.cc:826] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:23.994915 29721 tablet_bootstrap.cc:492] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:23.995687 29721 tablet_bootstrap.cc:492] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Bootstrap complete.
I20250809 02:06:23.997155 29721 ts_tablet_manager.cc:1397] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Time spent bootstrapping tablet: real 0.135s user 0.106s sys 0.025s
I20250809 02:06:24.018703 29721 raft_consensus.cc:357] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:24.022178 29721 raft_consensus.cc:738] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Initialized, Role: FOLLOWER
I20250809 02:06:24.023270 29721 consensus_queue.cc:260] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:24.024029 29721 raft_consensus.cc:397] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:24.024354 29721 raft_consensus.cc:491] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:24.024843 29721 raft_consensus.cc:3058] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:06:24.031823 29679 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:41085
I20250809 02:06:24.032006 29828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:41085 every 8 connection(s)
I20250809 02:06:24.033130 29721 raft_consensus.cc:513] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:24.033763 29721 leader_election.cc:304] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d01767213e461c8b872b0b1e439fc1; no voters:
I20250809 02:06:24.034240 29679 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 02:06:24.036137 29721 leader_election.cc:290] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250809 02:06:24.036744 29829 raft_consensus.cc:2802] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 02:06:24.044710 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 29679
I20250809 02:06:24.047036 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.194:38491
--local_ip_for_outbound_sockets=127.25.185.194
--tserver_master_addrs=127.25.185.254:43523
--webserver_port=41475
--webserver_interface=127.25.185.194
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:06:24.065646 29829 raft_consensus.cc:695] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 2 LEADER]: Becoming Leader. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Running, Role: LEADER
I20250809 02:06:24.066967 29829 consensus_queue.cc:237] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:24.072711 29721 ts_tablet_manager.cc:1428] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Time spent starting tablet: real 0.075s user 0.048s sys 0.015s
I20250809 02:06:24.087457 29830 heartbeater.cc:344] Connected to a master server at 127.25.185.254:43523
I20250809 02:06:24.087877 29830 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:24.088843 29830 heartbeater.cc:507] Master 127.25.185.254:43523 requested a full tablet report, sending...
I20250809 02:06:24.092499 29642 ts_manager.cc:194] Registered new tserver with Master: d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193:41085)
I20250809 02:06:24.095078 29642 catalog_manager.cc:5582] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "d8d01767213e461c8b872b0b1e439fc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } health_report { overall_health: HEALTHY } } }
I20250809 02:06:24.127403 29642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:55663
I20250809 02:06:24.130874 29830 heartbeater.cc:499] Master 127.25.185.254:43523 was elected leader, sending a full tablet report...
W20250809 02:06:24.366185 29837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:24.366750 29837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:24.367287 29837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:24.397027 29837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:24.397850 29837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.194
I20250809 02:06:24.431572 29837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.194:38491
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.185.194
--webserver_port=41475
--tserver_master_addrs=127.25.185.254:43523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.194
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:24.433053 29837 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:24.434571 29837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:24.447018 29848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:25.850332 29847 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 29837
W20250809 02:06:26.179690 29837 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.732s user 0.696s sys 1.034s
W20250809 02:06:24.447959 29849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:26.180228 29837 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.733s user 0.696s sys 1.034s
W20250809 02:06:26.182618 29851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:26.184859 29850 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1732 milliseconds
I20250809 02:06:26.184877 29837 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:26.186064 29837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:26.188097 29837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:26.189456 29837 hybrid_clock.cc:648] HybridClock initialized: now 1754705186189423 us; error 33 us; skew 500 ppm
I20250809 02:06:26.190227 29837 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:26.196077 29837 webserver.cc:489] Webserver started at http://127.25.185.194:41475/ using document root <none> and password file <none>
I20250809 02:06:26.196925 29837 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:26.197122 29837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:26.204675 29837 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.003s
I20250809 02:06:26.209163 29858 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:26.210147 29837 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250809 02:06:26.210444 29837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "14f06d05002142c48d6482f6b3cd0791"
format_stamp: "Formatted at 2025-08-09 02:06:11 on dist-test-slave-24ff"
I20250809 02:06:26.212335 29837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:26.278254 29837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:26.279877 29837 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:26.280304 29837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:26.282897 29837 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:26.288970 29865 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 02:06:26.296376 29837 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 02:06:26.296607 29837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250809 02:06:26.296866 29837 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 02:06:26.301409 29837 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 02:06:26.301663 29837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.002s
I20250809 02:06:26.302021 29865 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap starting.
I20250809 02:06:26.357753 29865 log.cc:826] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:26.452955 29865 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:26.453742 29865 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap complete.
I20250809 02:06:26.455483 29865 ts_tablet_manager.cc:1397] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Time spent bootstrapping tablet: real 0.154s user 0.113s sys 0.039s
I20250809 02:06:26.470674 29865 raft_consensus.cc:357] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } }
I20250809 02:06:26.473174 29865 raft_consensus.cc:738] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f06d05002142c48d6482f6b3cd0791, State: Initialized, Role: FOLLOWER
I20250809 02:06:26.474037 29865 consensus_queue.cc:260] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } }
I20250809 02:06:26.474503 29865 raft_consensus.cc:397] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:26.474846 29865 raft_consensus.cc:491] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:26.475150 29865 raft_consensus.cc:3058] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:06:26.481571 29865 raft_consensus.cc:513] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } }
I20250809 02:06:26.482210 29865 leader_election.cc:304] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14f06d05002142c48d6482f6b3cd0791; no voters:
I20250809 02:06:26.484694 29865 leader_election.cc:290] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250809 02:06:26.485041 29970 raft_consensus.cc:2802] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 02:06:26.488698 29970 raft_consensus.cc:695] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEADER]: Becoming Leader. State: Replica: 14f06d05002142c48d6482f6b3cd0791, State: Running, Role: LEADER
I20250809 02:06:26.489749 29970 consensus_queue.cc:237] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } }
I20250809 02:06:26.492062 29837 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.194:38491
I20250809 02:06:26.492447 29865 ts_tablet_manager.cc:1428] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Time spent starting tablet: real 0.037s user 0.031s sys 0.001s
I20250809 02:06:26.492362 29975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.194:38491 every 8 connection(s)
I20250809 02:06:26.494848 29837 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 02:06:26.495496 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 29837
I20250809 02:06:26.497432 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.195:39413
--local_ip_for_outbound_sockets=127.25.185.195
--tserver_master_addrs=127.25.185.254:43523
--webserver_port=42581
--webserver_interface=127.25.185.195
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:06:26.526892 29977 heartbeater.cc:344] Connected to a master server at 127.25.185.254:43523
I20250809 02:06:26.527390 29977 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:26.528570 29977 heartbeater.cc:507] Master 127.25.185.254:43523 requested a full tablet report, sending...
I20250809 02:06:26.532784 29642 ts_manager.cc:194] Registered new tserver with Master: 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491)
I20250809 02:06:26.534076 29642 catalog_manager.cc:5582] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 reported cstate change: term changed from 0 to 2, leader changed from <none> to 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194), VOTER 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) added. New cstate: current_term: 2 leader_uuid: "14f06d05002142c48d6482f6b3cd0791" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } health_report { overall_health: HEALTHY } } }
I20250809 02:06:26.548022 29642 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.194:34729
I20250809 02:06:26.554307 29977 heartbeater.cc:499] Master 127.25.185.254:43523 was elected leader, sending a full tablet report...
I20250809 02:06:26.563889 29928 consensus_queue.cc:237] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } }
I20250809 02:06:26.567780 29970 raft_consensus.cc:2953] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } } }
I20250809 02:06:26.575423 29627 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 97b1f8c5f1234b6b85687619bc22c978 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250809 02:06:26.578740 29642 catalog_manager.cc:5582] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 reported cstate change: config changed from index -1 to 10, NON_VOTER d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) added. New cstate: current_term: 2 leader_uuid: "14f06d05002142c48d6482f6b3cd0791" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250809 02:06:26.582276 29859 consensus_peers.cc:489] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 -> Peer d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193:41085): Couldn't send request to peer d8d01767213e461c8b872b0b1e439fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97b1f8c5f1234b6b85687619bc22c978. This is attempt 1: this message will repeat every 5th retry.
W20250809 02:06:26.587517 29642 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 97b1f8c5f1234b6b85687619bc22c978 with cas_config_opid_index 10: no extra replica candidate found for tablet 97b1f8c5f1234b6b85687619bc22c978 (table TestTable1 [id=03aaab86d51b43a3b06c26154cc9ee73]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (2) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250809 02:06:26.814428 29982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:26.814939 29982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:26.815392 29982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:26.845562 29982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:26.846349 29982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.195
I20250809 02:06:26.880695 29982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.195:39413
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.185.195
--webserver_port=42581
--tserver_master_addrs=127.25.185.254:43523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.195
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:26.881935 29982 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:26.883935 29982 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:26.896210 29995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:06:27.193333 30002 ts_tablet_manager.cc:927] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Initiating tablet copy from peer 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491)
I20250809 02:06:27.196682 30002 tablet_copy_client.cc:323] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.25.185.194:38491
I20250809 02:06:27.219350 29948 tablet_copy_service.cc:140] P 14f06d05002142c48d6482f6b3cd0791: Received BeginTabletCopySession request for tablet 97b1f8c5f1234b6b85687619bc22c978 from peer d8d01767213e461c8b872b0b1e439fc1 ({username='slave'} at 127.25.185.193:41311)
I20250809 02:06:27.219800 29948 tablet_copy_service.cc:161] P 14f06d05002142c48d6482f6b3cd0791: Beginning new tablet copy session on tablet 97b1f8c5f1234b6b85687619bc22c978 from peer d8d01767213e461c8b872b0b1e439fc1 at {username='slave'} at 127.25.185.193:41311: session id = d8d01767213e461c8b872b0b1e439fc1-97b1f8c5f1234b6b85687619bc22c978
I20250809 02:06:27.224220 29948 tablet_copy_source_session.cc:215] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Tablet Copy: opened 0 blocks and 1 log segments
I20250809 02:06:27.227375 30002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b1f8c5f1234b6b85687619bc22c978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:27.237222 30002 tablet_copy_client.cc:806] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: tablet copy: Starting download of 0 data blocks...
I20250809 02:06:27.237843 30002 tablet_copy_client.cc:670] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: tablet copy: Starting download of 1 WAL segments...
I20250809 02:06:27.241590 30002 tablet_copy_client.cc:538] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250809 02:06:27.246907 30002 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap starting.
I20250809 02:06:27.334479 30002 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:27.335108 30002 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap complete.
I20250809 02:06:27.335542 30002 ts_tablet_manager.cc:1397] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Time spent bootstrapping tablet: real 0.089s user 0.075s sys 0.012s
I20250809 02:06:27.337316 30002 raft_consensus.cc:357] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } }
I20250809 02:06:27.337707 30002 raft_consensus.cc:738] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Initialized, Role: LEARNER
I20250809 02:06:27.338131 30002 consensus_queue.cc:260] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } }
I20250809 02:06:27.341213 30002 ts_tablet_manager.cc:1428] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20250809 02:06:27.343604 29948 tablet_copy_service.cc:342] P 14f06d05002142c48d6482f6b3cd0791: Request end of tablet copy session d8d01767213e461c8b872b0b1e439fc1-97b1f8c5f1234b6b85687619bc22c978 received from {username='slave'} at 127.25.185.193:41311
I20250809 02:06:27.344233 29948 tablet_copy_service.cc:434] P 14f06d05002142c48d6482f6b3cd0791: ending tablet copy session d8d01767213e461c8b872b0b1e439fc1-97b1f8c5f1234b6b85687619bc22c978 on tablet 97b1f8c5f1234b6b85687619bc22c978 with peer d8d01767213e461c8b872b0b1e439fc1
I20250809 02:06:27.717227 29784 raft_consensus.cc:1215] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
W20250809 02:06:26.897261 29996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:28.152125 29998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:28.154120 29997 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1254 milliseconds
I20250809 02:06:28.154206 29982 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:28.155398 29982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:28.157626 29982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:28.158952 29982 hybrid_clock.cc:648] HybridClock initialized: now 1754705188158945 us; error 53 us; skew 500 ppm
I20250809 02:06:28.159714 29982 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:28.165948 29982 webserver.cc:489] Webserver started at http://127.25.185.195:42581/ using document root <none> and password file <none>
I20250809 02:06:28.166888 29982 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:28.167104 29982 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:28.174741 29982 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20250809 02:06:28.179132 30016 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:28.180225 29982 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250809 02:06:28.180528 29982 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3ffc0ee6c47f460cbfa594f7a260c685"
format_stamp: "Formatted at 2025-08-09 02:06:13 on dist-test-slave-24ff"
I20250809 02:06:28.182461 29982 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:28.258636 29982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:28.260116 29982 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:28.260520 29982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:28.263000 29982 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:28.268280 30023 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 02:06:28.275193 30024 raft_consensus.cc:1062] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: attempting to promote NON_VOTER d8d01767213e461c8b872b0b1e439fc1 to VOTER
I20250809 02:06:28.276432 30024 consensus_queue.cc:237] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:28.276715 29982 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 02:06:28.277009 29982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.001s
I20250809 02:06:28.277331 29982 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 02:06:28.280784 29784 raft_consensus.cc:1273] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 LEARNER]: Refusing update from remote peer 14f06d05002142c48d6482f6b3cd0791: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250809 02:06:28.281883 30025 consensus_queue.cc:1035] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250809 02:06:28.282918 29982 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 02:06:28.283113 29982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.001s sys 0.005s
I20250809 02:06:28.283866 30023 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap starting.
I20250809 02:06:28.291612 29784 raft_consensus.cc:2953] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:28.289896 30024 raft_consensus.cc:2953] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:28.305564 29642 catalog_manager.cc:5582] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 reported cstate change: config changed from index 10 to 11, d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "14f06d05002142c48d6482f6b3cd0791" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250809 02:06:28.358598 30023 log.cc:826] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:28.467135 29982 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.195:39413
I20250809 02:06:28.467213 30139 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.195:39413 every 8 connection(s)
I20250809 02:06:28.470541 29982 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 02:06:28.479300 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 29982
I20250809 02:06:28.506402 30140 heartbeater.cc:344] Connected to a master server at 127.25.185.254:43523
I20250809 02:06:28.506917 30140 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:28.508112 30140 heartbeater.cc:507] Master 127.25.185.254:43523 requested a full tablet report, sending...
I20250809 02:06:28.511790 29641 ts_manager.cc:194] Registered new tserver with Master: 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413)
I20250809 02:06:28.514912 29641 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.195:35877
I20250809 02:06:28.516988 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:06:28.520829 30023 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:28.521445 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:06:28.521556 30023 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap complete.
I20250809 02:06:28.522835 30023 ts_tablet_manager.cc:1397] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Time spent bootstrapping tablet: real 0.240s user 0.195s sys 0.033s
W20250809 02:06:28.523962 26343 ts_itest-base.cc:209] found only 0 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" }
I20250809 02:06:28.533295 30023 raft_consensus.cc:357] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } }
I20250809 02:06:28.535111 30023 raft_consensus.cc:738] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffc0ee6c47f460cbfa594f7a260c685, State: Initialized, Role: FOLLOWER
I20250809 02:06:28.535723 30023 consensus_queue.cc:260] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } }
I20250809 02:06:28.536147 30023 raft_consensus.cc:397] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:28.536389 30023 raft_consensus.cc:491] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:28.536660 30023 raft_consensus.cc:3058] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:06:28.539927 29928 consensus_queue.cc:237] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: NON_VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: true } }
I20250809 02:06:28.541805 30023 raft_consensus.cc:513] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } }
I20250809 02:06:28.542547 30023 leader_election.cc:304] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3ffc0ee6c47f460cbfa594f7a260c685; no voters:
I20250809 02:06:28.544965 30023 leader_election.cc:290] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250809 02:06:28.546916 30146 raft_consensus.cc:2802] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 02:06:28.547343 29784 raft_consensus.cc:1273] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Refusing update from remote peer 14f06d05002142c48d6482f6b3cd0791: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250809 02:06:28.549379 30025 consensus_queue.cc:1035] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250809 02:06:28.554276 30023 ts_tablet_manager.cc:1428] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Time spent starting tablet: real 0.031s user 0.022s sys 0.007s
I20250809 02:06:28.555074 30140 heartbeater.cc:499] Master 127.25.185.254:43523 was elected leader, sending a full tablet report...
I20250809 02:06:28.556556 30024 raft_consensus.cc:2953] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: NON_VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: true } } }
I20250809 02:06:28.558346 30146 raft_consensus.cc:695] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 LEADER]: Becoming Leader. State: Replica: 3ffc0ee6c47f460cbfa594f7a260c685, State: Running, Role: LEADER
I20250809 02:06:28.559247 30146 consensus_queue.cc:237] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } }
I20250809 02:06:28.559372 29783 raft_consensus.cc:2953] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: NON_VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: true } } }
W20250809 02:06:28.565891 29859 consensus_peers.cc:489] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 -> Peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413): Couldn't send request to peer 3ffc0ee6c47f460cbfa594f7a260c685. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 97b1f8c5f1234b6b85687619bc22c978. This is attempt 1: this message will repeat every 5th retry.
I20250809 02:06:28.566325 29627 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 97b1f8c5f1234b6b85687619bc22c978 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250809 02:06:28.569434 29641 catalog_manager.cc:5582] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 reported cstate change: config changed from index 11 to 12, NON_VOTER 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) added. New cstate: current_term: 2 leader_uuid: "14f06d05002142c48d6482f6b3cd0791" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: NON_VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250809 02:06:28.576437 29640 catalog_manager.cc:5582] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 reported cstate change: term changed from 0 to 2, leader changed from <none> to 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195), VOTER 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) added. New cstate: current_term: 2 leader_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } health_report { overall_health: HEALTHY } } }
I20250809 02:06:28.596575 30095 consensus_queue.cc:237] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } }
I20250809 02:06:28.600014 30150 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } }
I20250809 02:06:28.608701 29626 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 895362632c774235a3224e8af3008462 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250809 02:06:28.611725 30018 consensus_peers.cc:489] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 -> Peer 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491): Couldn't send request to peer 14f06d05002142c48d6482f6b3cd0791. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 895362632c774235a3224e8af3008462. This is attempt 1: this message will repeat every 5th retry.
I20250809 02:06:28.611961 29640 catalog_manager.cc:5582] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 reported cstate change: config changed from index -1 to 10, NON_VOTER 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) added. New cstate: current_term: 2 leader_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250809 02:06:28.621338 30095 consensus_queue.cc:237] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } }
I20250809 02:06:28.624511 30149 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } } }
W20250809 02:06:28.626528 30018 consensus_peers.cc:489] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 -> Peer 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491): Couldn't send request to peer 14f06d05002142c48d6482f6b3cd0791. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 895362632c774235a3224e8af3008462. This is attempt 1: this message will repeat every 5th retry.
I20250809 02:06:28.631323 29626 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 895362632c774235a3224e8af3008462 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250809 02:06:28.634002 29640 catalog_manager.cc:5582] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 reported cstate change: config changed from index 10 to 11, NON_VOTER d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) added. New cstate: current_term: 2 leader_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250809 02:06:28.634791 30017 consensus_peers.cc:489] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 -> Peer d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193:41085): Couldn't send request to peer d8d01767213e461c8b872b0b1e439fc1. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 895362632c774235a3224e8af3008462. This is attempt 1: this message will repeat every 5th retry.
I20250809 02:06:28.887427 29627 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 97b1f8c5f1234b6b85687619bc22c978 with cas_config_opid_index 10: aborting the task: latest config opid_index 12; task opid_index 10
I20250809 02:06:28.943554 30157 ts_tablet_manager.cc:927] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Initiating tablet copy from peer 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491)
I20250809 02:06:28.945510 30157 tablet_copy_client.cc:323] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: tablet copy: Beginning tablet copy session from remote peer at address 127.25.185.194:38491
I20250809 02:06:28.947000 29948 tablet_copy_service.cc:140] P 14f06d05002142c48d6482f6b3cd0791: Received BeginTabletCopySession request for tablet 97b1f8c5f1234b6b85687619bc22c978 from peer 3ffc0ee6c47f460cbfa594f7a260c685 ({username='slave'} at 127.25.185.195:58977)
I20250809 02:06:28.947384 29948 tablet_copy_service.cc:161] P 14f06d05002142c48d6482f6b3cd0791: Beginning new tablet copy session on tablet 97b1f8c5f1234b6b85687619bc22c978 from peer 3ffc0ee6c47f460cbfa594f7a260c685 at {username='slave'} at 127.25.185.195:58977: session id = 3ffc0ee6c47f460cbfa594f7a260c685-97b1f8c5f1234b6b85687619bc22c978
I20250809 02:06:28.951447 29948 tablet_copy_source_session.cc:215] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Tablet Copy: opened 0 blocks and 1 log segments
I20250809 02:06:28.954162 30157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97b1f8c5f1234b6b85687619bc22c978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:28.966471 30157 tablet_copy_client.cc:806] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: tablet copy: Starting download of 0 data blocks...
I20250809 02:06:28.967005 30157 tablet_copy_client.cc:670] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: tablet copy: Starting download of 1 WAL segments...
I20250809 02:06:28.970463 30157 tablet_copy_client.cc:538] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250809 02:06:28.976181 30157 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap starting.
I20250809 02:06:29.076212 30160 ts_tablet_manager.cc:927] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Initiating tablet copy from peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413)
I20250809 02:06:29.077953 30160 tablet_copy_client.cc:323] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: tablet copy: Beginning tablet copy session from remote peer at address 127.25.185.195:39413
I20250809 02:06:29.079731 30162 ts_tablet_manager.cc:927] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Initiating tablet copy from peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413)
I20250809 02:06:29.079751 30115 tablet_copy_service.cc:140] P 3ffc0ee6c47f460cbfa594f7a260c685: Received BeginTabletCopySession request for tablet 895362632c774235a3224e8af3008462 from peer 14f06d05002142c48d6482f6b3cd0791 ({username='slave'} at 127.25.185.194:41461)
I20250809 02:06:29.080370 30115 tablet_copy_service.cc:161] P 3ffc0ee6c47f460cbfa594f7a260c685: Beginning new tablet copy session on tablet 895362632c774235a3224e8af3008462 from peer 14f06d05002142c48d6482f6b3cd0791 at {username='slave'} at 127.25.185.194:41461: session id = 14f06d05002142c48d6482f6b3cd0791-895362632c774235a3224e8af3008462
I20250809 02:06:29.080618 30157 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:29.081315 30162 tablet_copy_client.cc:323] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: tablet copy: Beginning tablet copy session from remote peer at address 127.25.185.195:39413
I20250809 02:06:29.081377 30157 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap complete.
I20250809 02:06:29.081926 30157 ts_tablet_manager.cc:1397] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Time spent bootstrapping tablet: real 0.106s user 0.094s sys 0.009s
I20250809 02:06:29.084923 30157 raft_consensus.cc:357] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: NON_VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: true } }
I20250809 02:06:29.085618 30157 raft_consensus.cc:738] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 3ffc0ee6c47f460cbfa594f7a260c685, State: Initialized, Role: LEARNER
I20250809 02:06:29.086123 30157 consensus_queue.cc:260] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: NON_VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: true } }
I20250809 02:06:29.087574 30115 tablet_copy_source_session.cc:215] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Tablet Copy: opened 0 blocks and 1 log segments
I20250809 02:06:29.088008 30157 ts_tablet_manager.cc:1428] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Time spent starting tablet: real 0.006s user 0.004s sys 0.004s
I20250809 02:06:29.089540 29948 tablet_copy_service.cc:342] P 14f06d05002142c48d6482f6b3cd0791: Request end of tablet copy session 3ffc0ee6c47f460cbfa594f7a260c685-97b1f8c5f1234b6b85687619bc22c978 received from {username='slave'} at 127.25.185.195:58977
I20250809 02:06:29.089962 29948 tablet_copy_service.cc:434] P 14f06d05002142c48d6482f6b3cd0791: ending tablet copy session 3ffc0ee6c47f460cbfa594f7a260c685-97b1f8c5f1234b6b85687619bc22c978 on tablet 97b1f8c5f1234b6b85687619bc22c978 with peer 3ffc0ee6c47f460cbfa594f7a260c685
I20250809 02:06:29.091758 30160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 895362632c774235a3224e8af3008462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:29.092341 30115 tablet_copy_service.cc:140] P 3ffc0ee6c47f460cbfa594f7a260c685: Received BeginTabletCopySession request for tablet 895362632c774235a3224e8af3008462 from peer d8d01767213e461c8b872b0b1e439fc1 ({username='slave'} at 127.25.185.193:50235)
I20250809 02:06:29.092775 30115 tablet_copy_service.cc:161] P 3ffc0ee6c47f460cbfa594f7a260c685: Beginning new tablet copy session on tablet 895362632c774235a3224e8af3008462 from peer d8d01767213e461c8b872b0b1e439fc1 at {username='slave'} at 127.25.185.193:50235: session id = d8d01767213e461c8b872b0b1e439fc1-895362632c774235a3224e8af3008462
I20250809 02:06:29.099532 30115 tablet_copy_source_session.cc:215] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Tablet Copy: opened 0 blocks and 1 log segments
I20250809 02:06:29.102093 30162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 895362632c774235a3224e8af3008462. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:06:29.106528 30160 tablet_copy_client.cc:806] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: tablet copy: Starting download of 0 data blocks...
I20250809 02:06:29.107049 30160 tablet_copy_client.cc:670] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: tablet copy: Starting download of 1 WAL segments...
I20250809 02:06:29.110585 30160 tablet_copy_client.cc:538] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250809 02:06:29.111989 30162 tablet_copy_client.cc:806] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: tablet copy: Starting download of 0 data blocks...
I20250809 02:06:29.112486 30162 tablet_copy_client.cc:670] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: tablet copy: Starting download of 1 WAL segments...
I20250809 02:06:29.115599 30162 tablet_copy_client.cc:538] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250809 02:06:29.116389 30160 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap starting.
I20250809 02:06:29.120713 30162 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap starting.
I20250809 02:06:29.228884 30160 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:29.229501 30160 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap complete.
I20250809 02:06:29.229929 30160 ts_tablet_manager.cc:1397] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Time spent bootstrapping tablet: real 0.114s user 0.088s sys 0.024s
I20250809 02:06:29.231595 30160 raft_consensus.cc:357] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } }
I20250809 02:06:29.232056 30160 raft_consensus.cc:738] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 14f06d05002142c48d6482f6b3cd0791, State: Initialized, Role: LEARNER
I20250809 02:06:29.232462 30160 consensus_queue.cc:260] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } }
I20250809 02:06:29.233995 30160 ts_tablet_manager.cc:1428] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250809 02:06:29.234930 30162 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:29.235620 30162 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap complete.
I20250809 02:06:29.235610 30115 tablet_copy_service.cc:342] P 3ffc0ee6c47f460cbfa594f7a260c685: Request end of tablet copy session 14f06d05002142c48d6482f6b3cd0791-895362632c774235a3224e8af3008462 received from {username='slave'} at 127.25.185.194:41461
I20250809 02:06:29.235956 30115 tablet_copy_service.cc:434] P 3ffc0ee6c47f460cbfa594f7a260c685: ending tablet copy session 14f06d05002142c48d6482f6b3cd0791-895362632c774235a3224e8af3008462 on tablet 895362632c774235a3224e8af3008462 with peer 14f06d05002142c48d6482f6b3cd0791
I20250809 02:06:29.236176 30162 ts_tablet_manager.cc:1397] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Time spent bootstrapping tablet: real 0.116s user 0.099s sys 0.016s
I20250809 02:06:29.238258 30162 raft_consensus.cc:357] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } }
I20250809 02:06:29.238919 30162 raft_consensus.cc:738] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Initialized, Role: LEARNER
I20250809 02:06:29.239291 30162 consensus_queue.cc:260] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: NON_VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: true } }
I20250809 02:06:29.242168 30162 ts_tablet_manager.cc:1428] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20250809 02:06:29.243871 30115 tablet_copy_service.cc:342] P 3ffc0ee6c47f460cbfa594f7a260c685: Request end of tablet copy session d8d01767213e461c8b872b0b1e439fc1-895362632c774235a3224e8af3008462 received from {username='slave'} at 127.25.185.193:50235
I20250809 02:06:29.244249 30115 tablet_copy_service.cc:434] P 3ffc0ee6c47f460cbfa594f7a260c685: ending tablet copy session d8d01767213e461c8b872b0b1e439fc1-895362632c774235a3224e8af3008462 on tablet 895362632c774235a3224e8af3008462 with peer d8d01767213e461c8b872b0b1e439fc1
I20250809 02:06:29.528237 29783 raft_consensus.cc:1215] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250809 02:06:29.528651 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d8d01767213e461c8b872b0b1e439fc1 to finish bootstrapping
I20250809 02:06:29.549888 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 14f06d05002142c48d6482f6b3cd0791 to finish bootstrapping
I20250809 02:06:29.560281 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3ffc0ee6c47f460cbfa594f7a260c685 to finish bootstrapping
I20250809 02:06:29.565369 30095 raft_consensus.cc:1215] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250809 02:06:29.609030 29928 raft_consensus.cc:1215] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250809 02:06:29.826516 29908 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 02:06:29.829829 30075 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 02:06:29.838769 29764 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 02:06:29.960350 30169 raft_consensus.cc:1062] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: attempting to promote NON_VOTER d8d01767213e461c8b872b0b1e439fc1 to VOTER
I20250809 02:06:29.962507 30169 consensus_queue.cc:237] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:29.982034 29783 raft_consensus.cc:1273] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 LEARNER]: Refusing update from remote peer 3ffc0ee6c47f460cbfa594f7a260c685: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250809 02:06:29.983717 30168 consensus_queue.cc:1035] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250809 02:06:29.990640 29928 raft_consensus.cc:1273] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEARNER]: Refusing update from remote peer 3ffc0ee6c47f460cbfa594f7a260c685: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250809 02:06:29.992172 30169 consensus_queue.cc:1035] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250809 02:06:29.996192 30146 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:30.001363 29783 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:30.007426 29928 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEARNER]: Committing config change with OpId 2.12: config changed from index 11 to 12, d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:30.020462 29641 catalog_manager.cc:5582] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 reported cstate change: config changed from index 11 to 12, d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: NON_VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: true } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:30.051386 30168 raft_consensus.cc:1062] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: attempting to promote NON_VOTER 14f06d05002142c48d6482f6b3cd0791 to VOTER
I20250809 02:06:30.053436 30168 consensus_queue.cc:237] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:30.061555 29928 raft_consensus.cc:1273] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEARNER]: Refusing update from remote peer 3ffc0ee6c47f460cbfa594f7a260c685: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250809 02:06:30.063051 30169 consensus_queue.cc:1035] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250809 02:06:30.064695 29783 raft_consensus.cc:1273] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Refusing update from remote peer 3ffc0ee6c47f460cbfa594f7a260c685: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250809 02:06:30.069061 30168 consensus_queue.cc:1035] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250809 02:06:30.080840 29928 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:30.077713 30146 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:30.088229 29783 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:30.106897 29640 catalog_manager.cc:5582] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 reported cstate change: config changed from index 12 to 13, 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
bde1b90c638440a6acb697035e19e66e | 127.25.185.254:43523 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.25.185.212:43589 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
14f06d05002142c48d6482f6b3cd0791 | 127.25.185.194:38491 | HEALTHY | <none> | 1 | 0
3ffc0ee6c47f460cbfa594f7a260c685 | 127.25.185.195:39413 | HEALTHY | <none> | 1 | 0
d8d01767213e461c8b872b0b1e439fc1 | 127.25.185.193:41085 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.25.185.193 | experimental | 127.25.185.193:41085
local_ip_for_outbound_sockets | 127.25.185.194 | experimental | 127.25.185.194:38491
local_ip_for_outbound_sockets | 127.25.185.195 | experimental | 127.25.185.195:39413
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.25.185.193:41085
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.25.185.194:38491
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.25.185.195:39413
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.25.185.212:43589 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250809 02:06:30.133173 26343 log_verifier.cc:126] Checking tablet 895362632c774235a3224e8af3008462
I20250809 02:06:30.136781 30174 raft_consensus.cc:1062] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: attempting to promote NON_VOTER 3ffc0ee6c47f460cbfa594f7a260c685 to VOTER
I20250809 02:06:30.138234 30174 consensus_queue.cc:237] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
I20250809 02:06:30.143452 30095 raft_consensus.cc:1273] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 LEARNER]: Refusing update from remote peer 14f06d05002142c48d6482f6b3cd0791: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250809 02:06:30.144958 29783 raft_consensus.cc:1273] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Refusing update from remote peer 14f06d05002142c48d6482f6b3cd0791: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250809 02:06:30.144932 30206 consensus_queue.cc:1035] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250809 02:06:30.146854 30206 consensus_queue.cc:1035] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250809 02:06:30.154455 30174 raft_consensus.cc:2953] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } } }
I20250809 02:06:30.156690 30095 raft_consensus.cc:2953] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } } }
I20250809 02:06:30.163664 29783 raft_consensus.cc:2953] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } } }
I20250809 02:06:30.168740 29642 catalog_manager.cc:5582] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 reported cstate change: config changed from index 12 to 13, 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "14f06d05002142c48d6482f6b3cd0791" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250809 02:06:30.245643 26343 log_verifier.cc:177] Verified matching terms for 13 ops in tablet 895362632c774235a3224e8af3008462
I20250809 02:06:30.245997 26343 log_verifier.cc:126] Checking tablet 97b1f8c5f1234b6b85687619bc22c978
I20250809 02:06:30.337872 26343 log_verifier.cc:177] Verified matching terms for 13 ops in tablet 97b1f8c5f1234b6b85687619bc22c978
I20250809 02:06:30.338109 26343 log_verifier.cc:126] Checking tablet cc5e6748fb43459899958a80681b884a
I20250809 02:06:30.363336 26343 log_verifier.cc:177] Verified matching terms for 7 ops in tablet cc5e6748fb43459899958a80681b884a
I20250809 02:06:30.363669 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 29609
I20250809 02:06:30.387744 26343 minidump.cc:252] Setting minidump size limit to 20M
I20250809 02:06:30.389026 26343 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:30.390034 26343 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:30.400363 30214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:30.401661 30216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:30.400766 30213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:06:30.401970 26343 server_base.cc:1047] running on GCE node
I20250809 02:06:30.506551 26343 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250809 02:06:30.506914 26343 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250809 02:06:30.507122 26343 hybrid_clock.cc:648] HybridClock initialized: now 1754705190507103 us; error 0 us; skew 500 ppm
I20250809 02:06:30.508013 26343 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:30.511812 26343 webserver.cc:489] Webserver started at http://0.0.0.0:46661/ using document root <none> and password file <none>
I20250809 02:06:30.512964 26343 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:30.513217 26343 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:30.519771 26343 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20250809 02:06:30.524155 30221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:30.525117 26343 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250809 02:06:30.525453 26343 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bde1b90c638440a6acb697035e19e66e"
format_stamp: "Formatted at 2025-08-09 02:06:07 on dist-test-slave-24ff"
I20250809 02:06:30.527724 26343 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:30.544677 26343 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:30.546160 26343 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:30.546540 26343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:30.554582 26343 sys_catalog.cc:263] Verifying existing consensus state
W20250809 02:06:30.559000 26343 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.25.185.254:43523 exists but no master address supplied!
I20250809 02:06:30.560802 26343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Bootstrap starting.
I20250809 02:06:30.600234 26343 log.cc:826] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:30.661986 26343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:30.662793 26343 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Bootstrap complete.
I20250809 02:06:30.676299 26343 raft_consensus.cc:357] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:30.676856 26343 raft_consensus.cc:738] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde1b90c638440a6acb697035e19e66e, State: Initialized, Role: FOLLOWER
I20250809 02:06:30.677624 26343 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:30.678078 26343 raft_consensus.cc:397] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:30.678334 26343 raft_consensus.cc:491] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:30.678645 26343 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 3 FOLLOWER]: Advancing to term 4
I20250809 02:06:30.683826 26343 raft_consensus.cc:513] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:30.684522 26343 leader_election.cc:304] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bde1b90c638440a6acb697035e19e66e; no voters:
I20250809 02:06:30.685611 26343 leader_election.cc:290] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [CANDIDATE]: Term 4 election: Requested vote from peers
I20250809 02:06:30.685870 30228 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 4 FOLLOWER]: Leader election won for term 4
I20250809 02:06:30.687101 30228 raft_consensus.cc:695] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 4 LEADER]: Becoming Leader. State: Replica: bde1b90c638440a6acb697035e19e66e, State: Running, Role: LEADER
I20250809 02:06:30.687809 30228 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:30.695490 30230 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde1b90c638440a6acb697035e19e66e. Latest consensus state: current_term: 4 leader_uuid: "bde1b90c638440a6acb697035e19e66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } } }
I20250809 02:06:30.695689 30229 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "bde1b90c638440a6acb697035e19e66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } } }
I20250809 02:06:30.696013 30230 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: This master's current role is: LEADER
I20250809 02:06:30.696237 30229 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: This master's current role is: LEADER
I20250809 02:06:30.723649 26343 tablet_replica.cc:331] stopping tablet replica
I20250809 02:06:30.724339 26343 raft_consensus.cc:2241] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 4 LEADER]: Raft consensus shutting down.
I20250809 02:06:30.724860 26343 raft_consensus.cc:2270] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 4 FOLLOWER]: Raft consensus is shut down!
I20250809 02:06:30.727684 26343 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250809 02:06:30.728338 26343 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250809 02:06:30.752907 26343 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250809 02:06:31.200258 30140 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:43523 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:43523: connect: Connection refused (error 111)
W20250809 02:06:31.207655 29977 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:43523 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:43523: connect: Connection refused (error 111)
W20250809 02:06:31.208662 29830 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:43523 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:43523: connect: Connection refused (error 111)
I20250809 02:06:35.877530 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 29679
I20250809 02:06:35.903504 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 29837
I20250809 02:06:35.930352 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 29982
I20250809 02:06:35.962172 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:43523
--webserver_interface=127.25.185.254
--webserver_port=43033
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:43523 with env {}
W20250809 02:06:36.260850 30301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:36.261420 30301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:36.261832 30301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:36.292663 30301 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:06:36.292960 30301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:36.293175 30301 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:06:36.293370 30301 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:06:36.327965 30301 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:43523
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:43523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=43033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:36.329267 30301 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:36.330863 30301 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:36.341019 30307 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:36.341894 30308 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:37.745188 30306 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 30301
W20250809 02:06:38.116640 30306 kernel_stack_watchdog.cc:198] Thread 30301 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 02:06:38.121428 30301 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.780s user 0.597s sys 1.182s
W20250809 02:06:38.121721 30301 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.780s user 0.597s sys 1.182s
W20250809 02:06:38.123901 30309 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1781 milliseconds
W20250809 02:06:38.123988 30310 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:06:38.123929 30301 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:38.125351 30301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:38.127796 30301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:38.129109 30301 hybrid_clock.cc:648] HybridClock initialized: now 1754705198129059 us; error 55 us; skew 500 ppm
I20250809 02:06:38.129923 30301 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:38.135915 30301 webserver.cc:489] Webserver started at http://127.25.185.254:43033/ using document root <none> and password file <none>
I20250809 02:06:38.136775 30301 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:38.136973 30301 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:38.144517 30301 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250809 02:06:38.148996 30318 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:38.150038 30301 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250809 02:06:38.150359 30301 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "bde1b90c638440a6acb697035e19e66e"
format_stamp: "Formatted at 2025-08-09 02:06:07 on dist-test-slave-24ff"
I20250809 02:06:38.152344 30301 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:38.211958 30301 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:38.213444 30301 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:38.213886 30301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:38.283288 30301 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:43523
I20250809 02:06:38.283351 30369 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:43523 every 8 connection(s)
I20250809 02:06:38.286321 30301 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 02:06:38.295171 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 30301
I20250809 02:06:38.296265 30370 sys_catalog.cc:263] Verifying existing consensus state
I20250809 02:06:38.296911 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:41085
--local_ip_for_outbound_sockets=127.25.185.193
--tserver_master_addrs=127.25.185.254:43523
--webserver_port=43221
--webserver_interface=127.25.185.193
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:06:38.301040 30370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Bootstrap starting.
I20250809 02:06:38.313565 30370 log.cc:826] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:38.391770 30370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:38.392557 30370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Bootstrap complete.
I20250809 02:06:38.411424 30370 raft_consensus.cc:357] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:38.413501 30370 raft_consensus.cc:738] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: bde1b90c638440a6acb697035e19e66e, State: Initialized, Role: FOLLOWER
I20250809 02:06:38.414299 30370 consensus_queue.cc:260] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:38.414839 30370 raft_consensus.cc:397] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:38.415198 30370 raft_consensus.cc:491] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:38.415508 30370 raft_consensus.cc:3058] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 5 FOLLOWER]: Advancing to term 6
I20250809 02:06:38.420619 30370 raft_consensus.cc:513] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:38.421283 30370 leader_election.cc:304] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bde1b90c638440a6acb697035e19e66e; no voters:
I20250809 02:06:38.423396 30370 leader_election.cc:290] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [CANDIDATE]: Term 6 election: Requested vote from peers
I20250809 02:06:38.423799 30374 raft_consensus.cc:2802] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 6 FOLLOWER]: Leader election won for term 6
I20250809 02:06:38.427029 30374 raft_consensus.cc:695] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [term 6 LEADER]: Becoming Leader. State: Replica: bde1b90c638440a6acb697035e19e66e, State: Running, Role: LEADER
I20250809 02:06:38.427882 30374 consensus_queue.cc:237] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } }
I20250809 02:06:38.428501 30370 sys_catalog.cc:564] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:06:38.437197 30376 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: SysCatalogTable state changed. Reason: New leader bde1b90c638440a6acb697035e19e66e. Latest consensus state: current_term: 6 leader_uuid: "bde1b90c638440a6acb697035e19e66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } } }
I20250809 02:06:38.438376 30375 sys_catalog.cc:455] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "bde1b90c638440a6acb697035e19e66e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bde1b90c638440a6acb697035e19e66e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 43523 } } }
I20250809 02:06:38.438998 30375 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: This master's current role is: LEADER
I20250809 02:06:38.441236 30376 sys_catalog.cc:458] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e [sys.catalog]: This master's current role is: LEADER
I20250809 02:06:38.453022 30381 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:06:38.468546 30381 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=03aaab86d51b43a3b06c26154cc9ee73]
I20250809 02:06:38.470299 30381 catalog_manager.cc:671] Loaded metadata for table TestTable [id=d197aa1d02e4419c85981c7f38b06866]
I20250809 02:06:38.472061 30381 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=d2b6cd52815d4b65af37cab8eed4ea47]
I20250809 02:06:38.480882 30381 tablet_loader.cc:96] loaded metadata for tablet 895362632c774235a3224e8af3008462 (table TestTable [id=d197aa1d02e4419c85981c7f38b06866])
I20250809 02:06:38.482456 30381 tablet_loader.cc:96] loaded metadata for tablet 97b1f8c5f1234b6b85687619bc22c978 (table TestTable1 [id=03aaab86d51b43a3b06c26154cc9ee73])
I20250809 02:06:38.483743 30381 tablet_loader.cc:96] loaded metadata for tablet cc5e6748fb43459899958a80681b884a (table TestTable2 [id=d2b6cd52815d4b65af37cab8eed4ea47])
I20250809 02:06:38.485224 30381 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:06:38.490952 30381 catalog_manager.cc:1261] Loaded cluster ID: d7cf4e86ee38451fa2ceee0cdb5e54a4
I20250809 02:06:38.491283 30381 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:06:38.499835 30381 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:06:38.504438 30391 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250809 02:06:38.505873 30381 catalog_manager.cc:5966] T 00000000000000000000000000000000 P bde1b90c638440a6acb697035e19e66e: Loaded TSK: 0
I20250809 02:06:38.507516 30381 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250809 02:06:38.640848 30372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:38.641379 30372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:38.641891 30372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:38.673611 30372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:38.674511 30372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:06:38.709218 30372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:41085
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=43221
--tserver_master_addrs=127.25.185.254:43523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:38.710573 30372 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:38.712188 30372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:38.725435 30397 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:38.725962 30398 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:40.129256 30396 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 30372
W20250809 02:06:40.178505 30372 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.455s user 0.523s sys 0.931s
W20250809 02:06:40.179286 30372 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.456s user 0.523s sys 0.931s
W20250809 02:06:40.188493 30402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:06:40.188673 30372 server_base.cc:1047] running on GCE node
I20250809 02:06:40.190121 30372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:40.192764 30372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:40.194257 30372 hybrid_clock.cc:648] HybridClock initialized: now 1754705200194196 us; error 49 us; skew 500 ppm
I20250809 02:06:40.195304 30372 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:40.202910 30372 webserver.cc:489] Webserver started at http://127.25.185.193:43221/ using document root <none> and password file <none>
I20250809 02:06:40.204169 30372 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:40.204454 30372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:40.214720 30372 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20250809 02:06:40.219897 30407 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:40.221063 30372 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250809 02:06:40.221396 30372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d8d01767213e461c8b872b0b1e439fc1"
format_stamp: "Formatted at 2025-08-09 02:06:09 on dist-test-slave-24ff"
I20250809 02:06:40.223476 30372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:40.294026 30372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:40.295641 30372 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:40.296123 30372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:40.299378 30372 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:40.305749 30414 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250809 02:06:40.321596 30372 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250809 02:06:40.321842 30372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.018s user 0.001s sys 0.002s
I20250809 02:06:40.322142 30372 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250809 02:06:40.327585 30414 tablet_bootstrap.cc:492] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Bootstrap starting.
I20250809 02:06:40.338164 30372 ts_tablet_manager.cc:610] Registered 3 tablets
I20250809 02:06:40.338459 30372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.016s user 0.013s sys 0.000s
I20250809 02:06:40.398697 30414 log.cc:826] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:40.517489 30372 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:41085
I20250809 02:06:40.517760 30521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:41085 every 8 connection(s)
I20250809 02:06:40.521127 30372 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 02:06:40.522482 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 30372
I20250809 02:06:40.524413 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.194:38491
--local_ip_for_outbound_sockets=127.25.185.194
--tserver_master_addrs=127.25.185.254:43523
--webserver_port=41475
--webserver_interface=127.25.185.194
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:06:40.528893 30414 tablet_bootstrap.cc:492] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:40.530284 30414 tablet_bootstrap.cc:492] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Bootstrap complete.
I20250809 02:06:40.532931 30414 ts_tablet_manager.cc:1397] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Time spent bootstrapping tablet: real 0.206s user 0.171s sys 0.028s
I20250809 02:06:40.559677 30414 raft_consensus.cc:357] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:40.562817 30414 raft_consensus.cc:738] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Initialized, Role: FOLLOWER
I20250809 02:06:40.563819 30414 consensus_queue.cc:260] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:40.564502 30414 raft_consensus.cc:397] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:06:40.564877 30414 raft_consensus.cc:491] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:06:40.565313 30414 raft_consensus.cc:3058] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Advancing to term 3
I20250809 02:06:40.573700 30414 raft_consensus.cc:513] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:40.574630 30414 leader_election.cc:304] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8d01767213e461c8b872b0b1e439fc1; no voters:
I20250809 02:06:40.579875 30522 heartbeater.cc:344] Connected to a master server at 127.25.185.254:43523
I20250809 02:06:40.580400 30522 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:40.581626 30522 heartbeater.cc:507] Master 127.25.185.254:43523 requested a full tablet report, sending...
I20250809 02:06:40.586723 30335 ts_manager.cc:194] Registered new tserver with Master: d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193:41085)
I20250809 02:06:40.587337 30414 leader_election.cc:290] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250809 02:06:40.587738 30527 raft_consensus.cc:2802] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 3 FOLLOWER]: Leader election won for term 3
I20250809 02:06:40.592075 30335 catalog_manager.cc:5582] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) added, VOTER 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) added, VOTER d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:40.603310 30527 raft_consensus.cc:695] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [term 3 LEADER]: Becoming Leader. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Running, Role: LEADER
I20250809 02:06:40.604357 30527 consensus_queue.cc:237] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } }
I20250809 02:06:40.609366 30414 ts_tablet_manager.cc:1428] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1: Time spent starting tablet: real 0.076s user 0.047s sys 0.012s
I20250809 02:06:40.610208 30414 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap starting.
I20250809 02:06:40.654148 30335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:53817
I20250809 02:06:40.658998 30522 heartbeater.cc:499] Master 127.25.185.254:43523 was elected leader, sending a full tablet report...
I20250809 02:06:40.665812 30334 catalog_manager.cc:5582] T cc5e6748fb43459899958a80681b884a P d8d01767213e461c8b872b0b1e439fc1 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "d8d01767213e461c8b872b0b1e439fc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } health_report { overall_health: HEALTHY } } }
I20250809 02:06:40.786899 30414 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:40.787786 30414 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap complete.
I20250809 02:06:40.789361 30414 ts_tablet_manager.cc:1397] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Time spent bootstrapping tablet: real 0.179s user 0.156s sys 0.007s
I20250809 02:06:40.791554 30414 raft_consensus.cc:357] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
I20250809 02:06:40.792251 30414 raft_consensus.cc:738] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Initialized, Role: FOLLOWER
I20250809 02:06:40.792897 30414 consensus_queue.cc:260] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
I20250809 02:06:40.794646 30414 ts_tablet_manager.cc:1428] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250809 02:06:40.795315 30414 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap starting.
W20250809 02:06:40.903240 30523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:40.903769 30523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:40.904274 30523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:40.936404 30523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:40.937762 30523 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.194
I20250809 02:06:40.952105 30414 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:40.952924 30414 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Bootstrap complete.
I20250809 02:06:40.954416 30414 ts_tablet_manager.cc:1397] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Time spent bootstrapping tablet: real 0.159s user 0.147s sys 0.008s
I20250809 02:06:40.956403 30414 raft_consensus.cc:357] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:40.957053 30414 raft_consensus.cc:738] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Initialized, Role: FOLLOWER
I20250809 02:06:40.957633 30414 consensus_queue.cc:260] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:40.959477 30414 ts_tablet_manager.cc:1428] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250809 02:06:40.978441 30523 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.194:38491
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.185.194
--webserver_port=41475
--tserver_master_addrs=127.25.185.254:43523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.194
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:40.979764 30523 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:40.981384 30523 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:40.994364 30544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:06:42.098882 30550 raft_consensus.cc:491] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:06:42.099536 30550 raft_consensus.cc:513] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
W20250809 02:06:42.111121 30409 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.185.194:38491: connect: Connection refused (error 111)
I20250809 02:06:42.120817 30550 leader_election.cc:290] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491), 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413)
W20250809 02:06:42.130649 30408 leader_election.cc:336] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413): Network error: Client connection negotiation failed: client connection to 127.25.185.195:39413: connect: Connection refused (error 111)
W20250809 02:06:42.131386 30409 leader_election.cc:336] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491): Network error: Client connection negotiation failed: client connection to 127.25.185.194:38491: connect: Connection refused (error 111)
I20250809 02:06:42.131968 30409 leader_election.cc:304] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8d01767213e461c8b872b0b1e439fc1; no voters: 14f06d05002142c48d6482f6b3cd0791, 3ffc0ee6c47f460cbfa594f7a260c685
I20250809 02:06:42.133008 30550 raft_consensus.cc:2747] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250809 02:06:42.418668 30550 raft_consensus.cc:491] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:06:42.419353 30550 raft_consensus.cc:513] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:42.421841 30550 leader_election.cc:290] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413), 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491)
W20250809 02:06:42.434162 30408 leader_election.cc:336] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413): Network error: Client connection negotiation failed: client connection to 127.25.185.195:39413: connect: Connection refused (error 111)
W20250809 02:06:42.435106 30409 leader_election.cc:336] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491): Network error: Client connection negotiation failed: client connection to 127.25.185.194:38491: connect: Connection refused (error 111)
I20250809 02:06:42.435658 30409 leader_election.cc:304] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8d01767213e461c8b872b0b1e439fc1; no voters: 14f06d05002142c48d6482f6b3cd0791, 3ffc0ee6c47f460cbfa594f7a260c685
I20250809 02:06:42.437181 30550 raft_consensus.cc:2747] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250809 02:06:42.398226 30543 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 30523
W20250809 02:06:42.883837 30523 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.889s user 0.712s sys 1.165s
W20250809 02:06:40.995632 30545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:42.884406 30523 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.889s user 0.712s sys 1.165s
W20250809 02:06:42.887182 30547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:42.889395 30546 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1890 milliseconds
I20250809 02:06:42.889468 30523 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:06:42.890601 30523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:42.892647 30523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:42.893981 30523 hybrid_clock.cc:648] HybridClock initialized: now 1754705202893956 us; error 36 us; skew 500 ppm
I20250809 02:06:42.894778 30523 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:42.900626 30523 webserver.cc:489] Webserver started at http://127.25.185.194:41475/ using document root <none> and password file <none>
I20250809 02:06:42.901540 30523 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:42.901763 30523 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:06:42.909466 30523 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.005s
I20250809 02:06:42.913968 30558 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:42.914996 30523 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250809 02:06:42.915321 30523 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "14f06d05002142c48d6482f6b3cd0791"
format_stamp: "Formatted at 2025-08-09 02:06:11 on dist-test-slave-24ff"
I20250809 02:06:42.917300 30523 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:06:42.977021 30523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:42.978513 30523 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:42.978955 30523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:42.981940 30523 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:42.987879 30565 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250809 02:06:43.000388 30523 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250809 02:06:43.000648 30523 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.002s
I20250809 02:06:43.000927 30523 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250809 02:06:43.006644 30565 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap starting.
I20250809 02:06:43.009186 30523 ts_tablet_manager.cc:610] Registered 2 tablets
I20250809 02:06:43.009395 30523 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.006s sys 0.000s
I20250809 02:06:43.060442 30565 log.cc:826] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:43.170924 30523 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.194:38491
I20250809 02:06:43.171041 30672 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.194:38491 every 8 connection(s)
I20250809 02:06:43.174465 30523 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 02:06:43.175613 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 30523
I20250809 02:06:43.177618 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.195:39413
--local_ip_for_outbound_sockets=127.25.185.195
--tserver_master_addrs=127.25.185.254:43523
--webserver_port=42581
--webserver_interface=127.25.185.195
--builtin_ntp_servers=127.25.185.212:43589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250809 02:06:43.215804 30565 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:43.216550 30565 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap complete.
I20250809 02:06:43.218149 30565 ts_tablet_manager.cc:1397] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Time spent bootstrapping tablet: real 0.212s user 0.179s sys 0.028s
I20250809 02:06:43.221277 30673 heartbeater.cc:344] Connected to a master server at 127.25.185.254:43523
I20250809 02:06:43.221784 30673 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:43.223026 30673 heartbeater.cc:507] Master 127.25.185.254:43523 requested a full tablet report, sending...
I20250809 02:06:43.227347 30334 ts_manager.cc:194] Registered new tserver with Master: 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491)
I20250809 02:06:43.231711 30334 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.194:56267
I20250809 02:06:43.232601 30565 raft_consensus.cc:357] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
I20250809 02:06:43.234913 30565 raft_consensus.cc:738] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f06d05002142c48d6482f6b3cd0791, State: Initialized, Role: FOLLOWER
I20250809 02:06:43.235605 30565 consensus_queue.cc:260] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
I20250809 02:06:43.238580 30673 heartbeater.cc:499] Master 127.25.185.254:43523 was elected leader, sending a full tablet report...
I20250809 02:06:43.239123 30565 ts_tablet_manager.cc:1428] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791: Time spent starting tablet: real 0.021s user 0.016s sys 0.008s
I20250809 02:06:43.239761 30565 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap starting.
I20250809 02:06:43.355731 30565 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:43.356431 30565 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Bootstrap complete.
I20250809 02:06:43.357626 30565 ts_tablet_manager.cc:1397] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Time spent bootstrapping tablet: real 0.118s user 0.107s sys 0.008s
I20250809 02:06:43.359148 30565 raft_consensus.cc:357] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:43.359597 30565 raft_consensus.cc:738] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14f06d05002142c48d6482f6b3cd0791, State: Initialized, Role: FOLLOWER
I20250809 02:06:43.360100 30565 consensus_queue.cc:260] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:43.361454 30565 ts_tablet_manager.cc:1428] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250809 02:06:43.511037 30677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:06:43.511513 30677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:06:43.511950 30677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:06:43.551769 30677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:06:43.552593 30677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.195
I20250809 02:06:43.617841 30677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.195:39413
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.185.195
--webserver_port=42581
--tserver_master_addrs=127.25.185.254:43523
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.195
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:06:43.619935 30677 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:06:43.622592 30677 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:06:43.700206 30685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:06:43.907872 30689 raft_consensus.cc:491] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:06:43.908629 30689 raft_consensus.cc:513] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
I20250809 02:06:43.911495 30689 leader_election.cc:290] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491), 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413)
W20250809 02:06:43.950166 30408 leader_election.cc:336] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413): Network error: Client connection negotiation failed: client connection to 127.25.185.195:39413: connect: Connection refused (error 111)
I20250809 02:06:43.952898 30628 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97b1f8c5f1234b6b85687619bc22c978" candidate_uuid: "d8d01767213e461c8b872b0b1e439fc1" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "14f06d05002142c48d6482f6b3cd0791" is_pre_election: true
I20250809 02:06:43.953879 30628 raft_consensus.cc:2466] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d01767213e461c8b872b0b1e439fc1 in term 2.
I20250809 02:06:43.955441 30409 leader_election.cc:304] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14f06d05002142c48d6482f6b3cd0791, d8d01767213e461c8b872b0b1e439fc1; no voters: 3ffc0ee6c47f460cbfa594f7a260c685
I20250809 02:06:43.956065 30689 raft_consensus.cc:2802] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250809 02:06:43.956384 30689 raft_consensus.cc:491] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 02:06:43.956660 30689 raft_consensus.cc:3058] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Advancing to term 3
I20250809 02:06:43.961496 30689 raft_consensus.cc:513] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
I20250809 02:06:43.963223 30689 leader_election.cc:290] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 election: Requested vote from peers 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491), 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413)
I20250809 02:06:43.964140 30628 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97b1f8c5f1234b6b85687619bc22c978" candidate_uuid: "d8d01767213e461c8b872b0b1e439fc1" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "14f06d05002142c48d6482f6b3cd0791"
I20250809 02:06:43.964694 30628 raft_consensus.cc:3058] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Advancing to term 3
W20250809 02:06:43.968111 30408 leader_election.cc:336] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413): Network error: Client connection negotiation failed: client connection to 127.25.185.195:39413: connect: Connection refused (error 111)
I20250809 02:06:43.972314 30628 raft_consensus.cc:2466] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d01767213e461c8b872b0b1e439fc1 in term 3.
I20250809 02:06:43.973201 30409 leader_election.cc:304] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14f06d05002142c48d6482f6b3cd0791, d8d01767213e461c8b872b0b1e439fc1; no voters: 3ffc0ee6c47f460cbfa594f7a260c685
I20250809 02:06:43.973786 30689 raft_consensus.cc:2802] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 3 FOLLOWER]: Leader election won for term 3
I20250809 02:06:43.975749 30689 raft_consensus.cc:695] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [term 3 LEADER]: Becoming Leader. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Running, Role: LEADER
I20250809 02:06:43.976609 30689 consensus_queue.cc:237] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
I20250809 02:06:43.985548 30334 catalog_manager.cc:5582] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 reported cstate change: term changed from 2 to 3, leader changed from 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) to d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193). New cstate: current_term: 3 leader_uuid: "d8d01767213e461c8b872b0b1e439fc1" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250809 02:06:44.389556 30408 consensus_peers.cc:489] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 -> Peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413): Couldn't send request to peer 3ffc0ee6c47f460cbfa594f7a260c685. Status: Network error: Client connection negotiation failed: client connection to 127.25.185.195:39413: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250809 02:06:44.475147 30628 raft_consensus.cc:1273] T 97b1f8c5f1234b6b85687619bc22c978 P 14f06d05002142c48d6482f6b3cd0791 [term 3 FOLLOWER]: Refusing update from remote peer d8d01767213e461c8b872b0b1e439fc1: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250809 02:06:44.477474 30689 consensus_queue.cc:1035] T 97b1f8c5f1234b6b85687619bc22c978 P d8d01767213e461c8b872b0b1e439fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20250809 02:06:44.619580 30702 raft_consensus.cc:491] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:06:44.620268 30702 raft_consensus.cc:513] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:44.622886 30702 leader_election.cc:290] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413), 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491)
I20250809 02:06:44.625092 30628 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "895362632c774235a3224e8af3008462" candidate_uuid: "d8d01767213e461c8b872b0b1e439fc1" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "14f06d05002142c48d6482f6b3cd0791" is_pre_election: true
I20250809 02:06:44.625909 30628 raft_consensus.cc:2466] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8d01767213e461c8b872b0b1e439fc1 in term 2.
I20250809 02:06:44.627554 30409 leader_election.cc:304] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14f06d05002142c48d6482f6b3cd0791, d8d01767213e461c8b872b0b1e439fc1; no voters:
I20250809 02:06:44.628785 30702 raft_consensus.cc:2802] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250809 02:06:44.629287 30702 raft_consensus.cc:491] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 02:06:44.629781 30702 raft_consensus.cc:3058] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 2 FOLLOWER]: Advancing to term 3
W20250809 02:06:44.629727 30408 leader_election.cc:336] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413): Network error: Client connection negotiation failed: client connection to 127.25.185.195:39413: connect: Connection refused (error 111)
I20250809 02:06:44.639516 30702 raft_consensus.cc:513] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:44.643537 30628 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "895362632c774235a3224e8af3008462" candidate_uuid: "d8d01767213e461c8b872b0b1e439fc1" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "14f06d05002142c48d6482f6b3cd0791"
I20250809 02:06:44.644326 30628 raft_consensus.cc:3058] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 2 FOLLOWER]: Advancing to term 3
W20250809 02:06:44.648761 30408 leader_election.cc:336] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413): Network error: Client connection negotiation failed: client connection to 127.25.185.195:39413: connect: Connection refused (error 111)
I20250809 02:06:44.649803 30702 leader_election.cc:290] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 election: Requested vote from peers 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413), 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491)
I20250809 02:06:44.653641 30628 raft_consensus.cc:2466] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8d01767213e461c8b872b0b1e439fc1 in term 3.
I20250809 02:06:44.655172 30409 leader_election.cc:304] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 14f06d05002142c48d6482f6b3cd0791, d8d01767213e461c8b872b0b1e439fc1; no voters: 3ffc0ee6c47f460cbfa594f7a260c685
I20250809 02:06:44.656265 30702 raft_consensus.cc:2802] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 3 FOLLOWER]: Leader election won for term 3
I20250809 02:06:44.657217 30702 raft_consensus.cc:695] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 3 LEADER]: Becoming Leader. State: Replica: d8d01767213e461c8b872b0b1e439fc1, State: Running, Role: LEADER
I20250809 02:06:44.658469 30702 consensus_queue.cc:237] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:44.671169 30335 catalog_manager.cc:5582] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 reported cstate change: term changed from 2 to 3, leader changed from <none> to d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193). New cstate: current_term: 3 leader_uuid: "d8d01767213e461c8b872b0b1e439fc1" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250809 02:06:44.722976 30518 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5120 after lost signal to thread 30392
W20250809 02:06:44.724577 30518 debug-util.cc:398] Leaking SignalData structure 0x7b08000b5380 after lost signal to thread 30521
I20250809 02:06:45.104440 30628 raft_consensus.cc:1273] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 3 FOLLOWER]: Refusing update from remote peer d8d01767213e461c8b872b0b1e439fc1: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250809 02:06:45.106791 30702 consensus_queue.cc:1035] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20250809 02:06:45.099531 30684 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 30677
W20250809 02:06:45.160806 30684 kernel_stack_watchdog.cc:198] Thread 30677 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 02:06:43.704664 30686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:06:45.161825 30677 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.464s user 0.540s sys 0.909s
W20250809 02:06:45.163479 30408 consensus_peers.cc:489] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 -> Peer 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413): Couldn't send request to peer 3ffc0ee6c47f460cbfa594f7a260c685. Status: Network error: Client connection negotiation failed: client connection to 127.25.185.195:39413: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250809 02:06:45.178629 30677 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.481s user 0.541s sys 0.909s
I20250809 02:06:45.210036 30677 server_base.cc:1047] running on GCE node
W20250809 02:06:45.212314 30715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:06:45.213810 30677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:06:45.223676 30677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:06:45.225234 30677 hybrid_clock.cc:648] HybridClock initialized: now 1754705205225177 us; error 44 us; skew 500 ppm
I20250809 02:06:45.226437 30677 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:06:45.226306 30477 consensus_queue.cc:237] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:45.231878 30627 raft_consensus.cc:1273] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 3 FOLLOWER]: Refusing update from remote peer d8d01767213e461c8b872b0b1e439fc1: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250809 02:06:45.233280 30689 consensus_queue.cc:1035] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250809 02:06:45.239763 30689 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:45.242231 30627 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:45.258219 30319 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 895362632c774235a3224e8af3008462 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250809 02:06:45.264779 30335 catalog_manager.cc:5582] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 reported cstate change: config changed from index 13 to 15, VOTER 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195) evicted. New cstate: current_term: 3 leader_uuid: "d8d01767213e461c8b872b0b1e439fc1" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250809 02:06:45.274195 30677 webserver.cc:489] Webserver started at http://127.25.185.195:42581/ using document root <none> and password file <none>
I20250809 02:06:45.275683 30677 fs_manager.cc:362] Metadata directory not provided
I20250809 02:06:45.276021 30677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250809 02:06:45.283638 30335 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 895362632c774235a3224e8af3008462 on TS 3ffc0ee6c47f460cbfa594f7a260c685: Not found: failed to reset TS proxy: Could not find TS for UUID 3ffc0ee6c47f460cbfa594f7a260c685
I20250809 02:06:45.288887 30477 consensus_queue.cc:237] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:45.291563 30689 raft_consensus.cc:2953] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } }
I20250809 02:06:45.301612 30319 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 895362632c774235a3224e8af3008462 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250809 02:06:45.304980 30677 fs_manager.cc:714] Time spent opening directory manager: real 0.025s user 0.006s sys 0.018s
W20250809 02:06:45.307044 30319 catalog_manager.cc:5105] ChangeConfig:REMOVE_PEER RPC for tablet 895362632c774235a3224e8af3008462 with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER failed with leader d8d01767213e461c8b872b0b1e439fc1 (127.25.185.193:41085) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 15 but the committed config has opid_index of 16
I20250809 02:06:45.313290 30335 catalog_manager.cc:5582] T 895362632c774235a3224e8af3008462 P d8d01767213e461c8b872b0b1e439fc1 reported cstate change: config changed from index 15 to 16, VOTER 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194) evicted. New cstate: current_term: 3 leader_uuid: "d8d01767213e461c8b872b0b1e439fc1" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250809 02:06:45.317057 30724 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:06:45.318634 30677 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.000s sys 0.008s
I20250809 02:06:45.319139 30677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3ffc0ee6c47f460cbfa594f7a260c685"
format_stamp: "Formatted at 2025-08-09 02:06:13 on dist-test-slave-24ff"
I20250809 02:06:45.322229 30677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250809 02:06:45.337714 30320 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 895362632c774235a3224e8af3008462 on TS 3ffc0ee6c47f460cbfa594f7a260c685 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 3ffc0ee6c47f460cbfa594f7a260c685
I20250809 02:06:45.354089 30608 tablet_service.cc:1515] Processing DeleteTablet for tablet 895362632c774235a3224e8af3008462 with delete_type TABLET_DATA_TOMBSTONED (TS 14f06d05002142c48d6482f6b3cd0791 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:33198
I20250809 02:06:45.369760 30726 tablet_replica.cc:331] stopping tablet replica
I20250809 02:06:45.370872 30726 raft_consensus.cc:2241] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250809 02:06:45.371627 30726 raft_consensus.cc:2270] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250809 02:06:45.375555 30726 ts_tablet_manager.cc:1905] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250809 02:06:45.391057 30726 ts_tablet_manager.cc:1918] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250809 02:06:45.391687 30726 log.cc:1199] T 895362632c774235a3224e8af3008462 P 14f06d05002142c48d6482f6b3cd0791: Deleting WAL directory at /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/wals/895362632c774235a3224e8af3008462
I20250809 02:06:45.393292 30320 catalog_manager.cc:4928] TS 14f06d05002142c48d6482f6b3cd0791 (127.25.185.194:38491): tablet 895362632c774235a3224e8af3008462 (table TestTable [id=d197aa1d02e4419c85981c7f38b06866]) successfully deleted
I20250809 02:06:45.410953 30677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:06:45.412408 30677 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:06:45.412873 30677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:06:45.415608 30677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:06:45.421670 30733 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250809 02:06:45.441578 30677 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250809 02:06:45.441928 30677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.022s user 0.001s sys 0.001s
I20250809 02:06:45.442303 30677 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250809 02:06:45.451890 30733 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap starting.
I20250809 02:06:45.457630 30677 ts_tablet_manager.cc:610] Registered 2 tablets
I20250809 02:06:45.457929 30677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.016s user 0.010s sys 0.002s
I20250809 02:06:45.514945 30733 log.cc:826] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Log is configured to *not* fsync() on all Append() calls
I20250809 02:06:45.668033 30733 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:45.669267 30733 tablet_bootstrap.cc:492] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap complete.
I20250809 02:06:45.670396 30677 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.195:39413
I20250809 02:06:45.670658 30840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.195:39413 every 8 connection(s)
I20250809 02:06:45.671392 30733 ts_tablet_manager.cc:1397] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Time spent bootstrapping tablet: real 0.220s user 0.182s sys 0.030s
I20250809 02:06:45.674319 30677 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 02:06:45.685031 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 30677
I20250809 02:06:45.706665 30733 raft_consensus.cc:357] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
I20250809 02:06:45.711869 30733 raft_consensus.cc:738] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffc0ee6c47f460cbfa594f7a260c685, State: Initialized, Role: FOLLOWER
I20250809 02:06:45.712931 30733 consensus_queue.cc:260] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } } peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } attrs { promote: false } }
I20250809 02:06:45.717237 30733 ts_tablet_manager.cc:1428] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685: Time spent starting tablet: real 0.046s user 0.028s sys 0.004s
I20250809 02:06:45.718219 30733 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap starting.
I20250809 02:06:45.739111 30841 heartbeater.cc:344] Connected to a master server at 127.25.185.254:43523
I20250809 02:06:45.739667 30841 heartbeater.cc:461] Registering TS with master...
I20250809 02:06:45.741401 30841 heartbeater.cc:507] Master 127.25.185.254:43523 requested a full tablet report, sending...
I20250809 02:06:45.747923 30335 ts_manager.cc:194] Registered new tserver with Master: 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413)
I20250809 02:06:45.753918 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:06:45.754274 30335 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.195:48173
I20250809 02:06:45.759229 30841 heartbeater.cc:499] Master 127.25.185.254:43523 was elected leader, sending a full tablet report...
I20250809 02:06:45.759495 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250809 02:06:45.763557 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250809 02:06:45.841084 30733 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:06:45.841822 30733 tablet_bootstrap.cc:492] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Bootstrap complete.
I20250809 02:06:45.842993 30733 ts_tablet_manager.cc:1397] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Time spent bootstrapping tablet: real 0.125s user 0.106s sys 0.016s
I20250809 02:06:45.844533 30733 raft_consensus.cc:357] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:45.845005 30733 raft_consensus.cc:738] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ffc0ee6c47f460cbfa594f7a260c685, State: Initialized, Role: FOLLOWER
I20250809 02:06:45.845463 30733 consensus_queue.cc:260] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "3ffc0ee6c47f460cbfa594f7a260c685" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 39413 } } peers { permanent_uuid: "14f06d05002142c48d6482f6b3cd0791" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 38491 } attrs { promote: false } } peers { permanent_uuid: "d8d01767213e461c8b872b0b1e439fc1" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 41085 } attrs { promote: false } }
I20250809 02:06:45.847054 30733 ts_tablet_manager.cc:1428] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250809 02:06:45.967043 30776 tablet_service.cc:1515] Processing DeleteTablet for tablet 895362632c774235a3224e8af3008462 with delete_type TABLET_DATA_TOMBSTONED (TS 3ffc0ee6c47f460cbfa594f7a260c685 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:55618
I20250809 02:06:45.972327 30851 tablet_replica.cc:331] stopping tablet replica
I20250809 02:06:45.972992 30851 raft_consensus.cc:2241] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250809 02:06:45.973412 30851 raft_consensus.cc:2270] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250809 02:06:45.976398 30851 ts_tablet_manager.cc:1905] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250809 02:06:45.990860 30796 raft_consensus.cc:3058] T 97b1f8c5f1234b6b85687619bc22c978 P 3ffc0ee6c47f460cbfa594f7a260c685 [term 2 FOLLOWER]: Advancing to term 3
I20250809 02:06:45.992347 30851 ts_tablet_manager.cc:1918] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250809 02:06:45.992789 30851 log.cc:1199] T 895362632c774235a3224e8af3008462 P 3ffc0ee6c47f460cbfa594f7a260c685: Deleting WAL directory at /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/wals/895362632c774235a3224e8af3008462
I20250809 02:06:45.994820 30319 catalog_manager.cc:4928] TS 3ffc0ee6c47f460cbfa594f7a260c685 (127.25.185.195:39413): tablet 895362632c774235a3224e8af3008462 (table TestTable [id=d197aa1d02e4419c85981c7f38b06866]) successfully deleted
W20250809 02:06:46.767719 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:47.771090 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:48.775223 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:49.779009 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:50.782451 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:51.786018 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:52.789498 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:53.792829 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:54.796423 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:55.799777 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:56.803579 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:57.806907 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:58.811075 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:06:59.814759 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:07:00.818076 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:07:01.821322 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:07:02.824729 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:07:03.828042 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250809 02:07:04.831259 26343 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 895362632c774235a3224e8af3008462: tablet_id: "895362632c774235a3224e8af3008462" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250809 02:07:05.834198 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 30372
I20250809 02:07:05.862913 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 30523
I20250809 02:07:05.890420 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 30677
I20250809 02:07:05.917397 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 30301
2025-08-09T02:07:05Z chronyd exiting
I20250809 02:07:05.966472 26343 test_util.cc:183] -----------------------------------------------
I20250809 02:07:05.966709 26343 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754705097773638-26343-0
[ FAILED ] AdminCliTest.TestRebuildTables (60083 ms)
[----------] 5 tests from AdminCliTest (128124 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250809 02:07:05.970528 26343 test_util.cc:276] Using random seed: 632157000
I20250809 02:07:05.974391 26343 ts_itest-base.cc:115] Starting cluster with:
I20250809 02:07:05.974537 26343 ts_itest-base.cc:116] --------------
I20250809 02:07:05.974696 26343 ts_itest-base.cc:117] 5 tablet servers
I20250809 02:07:05.974830 26343 ts_itest-base.cc:118] 3 replicas per TS
I20250809 02:07:05.974957 26343 ts_itest-base.cc:119] --------------
2025-08-09T02:07:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T02:07:05Z Disabled control of system clock
I20250809 02:07:06.017727 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:36859
--webserver_interface=127.25.185.254
--webserver_port=0
--builtin_ntp_servers=127.25.185.212:37103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:36859
--raft_prepare_replacement_before_eviction=true with env {}
W20250809 02:07:06.317689 30875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:06.318312 30875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:06.318840 30875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:06.349665 30875 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 02:07:06.350112 30875 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:07:06.350366 30875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:06.350595 30875 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:07:06.350842 30875 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:07:06.385208 30875 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:37103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:36859
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:36859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:06.386415 30875 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:06.387987 30875 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:06.397573 30881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:06.399207 30882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:07.802186 30880 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 30875
W20250809 02:07:08.180725 30880 kernel_stack_watchdog.cc:198] Thread 30875 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 02:07:08.181188 30875 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.783s user 0.001s sys 0.000s
W20250809 02:07:08.181481 30875 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.783s user 0.001s sys 0.000s
W20250809 02:07:08.184916 30883 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1782 milliseconds
W20250809 02:07:08.186213 30884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:08.186296 30875 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:08.187536 30875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:08.191330 30875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:08.192791 30875 hybrid_clock.cc:648] HybridClock initialized: now 1754705228192742 us; error 64 us; skew 500 ppm
I20250809 02:07:08.193603 30875 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:08.200426 30875 webserver.cc:489] Webserver started at http://127.25.185.254:42811/ using document root <none> and password file <none>
I20250809 02:07:08.201419 30875 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:08.201633 30875 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:08.202113 30875 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:08.206534 30875 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "928cd73d4077466abf0f18f30257156d"
format_stamp: "Formatted at 2025-08-09 02:07:08 on dist-test-slave-24ff"
I20250809 02:07:08.207764 30875 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "928cd73d4077466abf0f18f30257156d"
format_stamp: "Formatted at 2025-08-09 02:07:08 on dist-test-slave-24ff"
I20250809 02:07:08.214991 30875 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250809 02:07:08.220429 30892 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:08.221422 30875 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250809 02:07:08.221740 30875 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "928cd73d4077466abf0f18f30257156d"
format_stamp: "Formatted at 2025-08-09 02:07:08 on dist-test-slave-24ff"
I20250809 02:07:08.222079 30875 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:08.292469 30875 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:08.294373 30875 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:08.294867 30875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:08.364923 30875 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:36859
I20250809 02:07:08.364982 30943 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:36859 every 8 connection(s)
I20250809 02:07:08.367700 30875 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 02:07:08.368655 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 30875
I20250809 02:07:08.369189 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 02:07:08.374969 30944 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:08.398303 30944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d: Bootstrap starting.
I20250809 02:07:08.403492 30944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:08.405330 30944 log.cc:826] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:08.409734 30944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d: No bootstrap required, opened a new log
I20250809 02:07:08.427047 30944 raft_consensus.cc:357] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928cd73d4077466abf0f18f30257156d" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36859 } }
I20250809 02:07:08.427765 30944 raft_consensus.cc:383] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:08.428051 30944 raft_consensus.cc:738] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 928cd73d4077466abf0f18f30257156d, State: Initialized, Role: FOLLOWER
I20250809 02:07:08.428752 30944 consensus_queue.cc:260] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928cd73d4077466abf0f18f30257156d" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36859 } }
I20250809 02:07:08.429278 30944 raft_consensus.cc:397] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:07:08.429551 30944 raft_consensus.cc:491] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:07:08.429883 30944 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:08.433969 30944 raft_consensus.cc:513] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928cd73d4077466abf0f18f30257156d" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36859 } }
I20250809 02:07:08.434721 30944 leader_election.cc:304] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 928cd73d4077466abf0f18f30257156d; no voters:
I20250809 02:07:08.436522 30944 leader_election.cc:290] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:07:08.437400 30949 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:07:08.439473 30949 raft_consensus.cc:695] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [term 1 LEADER]: Becoming Leader. State: Replica: 928cd73d4077466abf0f18f30257156d, State: Running, Role: LEADER
I20250809 02:07:08.440234 30949 consensus_queue.cc:237] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928cd73d4077466abf0f18f30257156d" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36859 } }
I20250809 02:07:08.440943 30944 sys_catalog.cc:564] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:07:08.449450 30950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "928cd73d4077466abf0f18f30257156d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928cd73d4077466abf0f18f30257156d" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36859 } } }
I20250809 02:07:08.451385 30950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [sys.catalog]: This master's current role is: LEADER
I20250809 02:07:08.451006 30951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 928cd73d4077466abf0f18f30257156d. Latest consensus state: current_term: 1 leader_uuid: "928cd73d4077466abf0f18f30257156d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "928cd73d4077466abf0f18f30257156d" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 36859 } } }
I20250809 02:07:08.451874 30951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d [sys.catalog]: This master's current role is: LEADER
I20250809 02:07:08.454502 30957 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:07:08.465657 30957 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:07:08.483927 30957 catalog_manager.cc:1349] Generated new cluster ID: 8200c9dc836642b896035daf83785386
I20250809 02:07:08.484213 30957 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:07:08.500051 30957 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 02:07:08.501427 30957 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:07:08.514631 30957 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 928cd73d4077466abf0f18f30257156d: Generated new TSK 0
I20250809 02:07:08.515612 30957 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 02:07:08.533327 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:0
--local_ip_for_outbound_sockets=127.25.185.193
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36859
--builtin_ntp_servers=127.25.185.212:37103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250809 02:07:08.840323 30968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:08.840829 30968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:08.841305 30968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:08.872076 30968 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 02:07:08.872540 30968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:08.873329 30968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:07:08.908753 30968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:37103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36859
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:08.910063 30968 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:08.911630 30968 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:08.924530 30974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:10.325616 30973 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 30968
W20250809 02:07:10.398509 30973 kernel_stack_watchdog.cc:198] Thread 30968 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 398ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 02:07:10.398581 30968 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.473s user 0.431s sys 1.041s
W20250809 02:07:10.398993 30968 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.474s user 0.431s sys 1.041s
W20250809 02:07:08.925733 30975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:10.400424 30976 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1473 milliseconds
I20250809 02:07:10.400802 30968 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250809 02:07:10.400840 30977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:10.405093 30968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:10.407670 30968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:10.409088 30968 hybrid_clock.cc:648] HybridClock initialized: now 1754705230409024 us; error 70 us; skew 500 ppm
I20250809 02:07:10.409788 30968 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:10.416198 30968 webserver.cc:489] Webserver started at http://127.25.185.193:36291/ using document root <none> and password file <none>
I20250809 02:07:10.417022 30968 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:10.417187 30968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:10.417537 30968 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:10.421721 30968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "590350e6d4044505a84578f3c330b110"
format_stamp: "Formatted at 2025-08-09 02:07:10 on dist-test-slave-24ff"
I20250809 02:07:10.422787 30968 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "590350e6d4044505a84578f3c330b110"
format_stamp: "Formatted at 2025-08-09 02:07:10 on dist-test-slave-24ff"
I20250809 02:07:10.430227 30968 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250809 02:07:10.436203 30984 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:10.437290 30968 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250809 02:07:10.437556 30968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "590350e6d4044505a84578f3c330b110"
format_stamp: "Formatted at 2025-08-09 02:07:10 on dist-test-slave-24ff"
I20250809 02:07:10.437857 30968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:10.487470 30968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:10.488865 30968 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:10.489244 30968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:10.492138 30968 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:10.496186 30968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:07:10.496358 30968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:10.496529 30968 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:07:10.496652 30968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:10.656498 30968 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:40993
I20250809 02:07:10.656613 31096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:40993 every 8 connection(s)
I20250809 02:07:10.659222 30968 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 02:07:10.669008 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 30968
I20250809 02:07:10.669517 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 02:07:10.676704 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.194:0
--local_ip_for_outbound_sockets=127.25.185.194
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36859
--builtin_ntp_servers=127.25.185.212:37103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250809 02:07:10.684267 31097 heartbeater.cc:344] Connected to a master server at 127.25.185.254:36859
I20250809 02:07:10.684820 31097 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:10.686111 31097 heartbeater.cc:507] Master 127.25.185.254:36859 requested a full tablet report, sending...
I20250809 02:07:10.688971 30909 ts_manager.cc:194] Registered new tserver with Master: 590350e6d4044505a84578f3c330b110 (127.25.185.193:40993)
I20250809 02:07:10.690882 30909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:36293
W20250809 02:07:11.001385 31101 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:11.001893 31101 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:11.002374 31101 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:11.033177 31101 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 02:07:11.033602 31101 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:11.034405 31101 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.194
I20250809 02:07:11.070827 31101 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:37103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36859
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.194
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:11.072124 31101 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:11.073813 31101 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:11.084856 31107 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:11.693972 31097 heartbeater.cc:499] Master 127.25.185.254:36859 was elected leader, sending a full tablet report...
W20250809 02:07:11.086719 31108 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:11.089684 31110 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:12.222505 31109 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250809 02:07:12.222597 31101 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:12.226392 31101 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:12.229105 31101 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:12.230515 31101 hybrid_clock.cc:648] HybridClock initialized: now 1754705232230479 us; error 60 us; skew 500 ppm
I20250809 02:07:12.231319 31101 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:12.237684 31101 webserver.cc:489] Webserver started at http://127.25.185.194:38829/ using document root <none> and password file <none>
I20250809 02:07:12.238608 31101 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:12.238898 31101 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:12.239516 31101 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:12.243953 31101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "25cde50f92a443ae86471232863f6a3b"
format_stamp: "Formatted at 2025-08-09 02:07:12 on dist-test-slave-24ff"
I20250809 02:07:12.245077 31101 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "25cde50f92a443ae86471232863f6a3b"
format_stamp: "Formatted at 2025-08-09 02:07:12 on dist-test-slave-24ff"
I20250809 02:07:12.251962 31101 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250809 02:07:12.257705 31117 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:12.258731 31101 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250809 02:07:12.259058 31101 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "25cde50f92a443ae86471232863f6a3b"
format_stamp: "Formatted at 2025-08-09 02:07:12 on dist-test-slave-24ff"
I20250809 02:07:12.259379 31101 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:12.309114 31101 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:12.310540 31101 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:12.311036 31101 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:12.313527 31101 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:12.317730 31101 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:07:12.317965 31101 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:12.318223 31101 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:07:12.318382 31101 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:12.453754 31101 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.194:40035
I20250809 02:07:12.453930 31229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.194:40035 every 8 connection(s)
I20250809 02:07:12.456584 31101 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250809 02:07:12.460017 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 31101
I20250809 02:07:12.460557 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250809 02:07:12.467671 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.195:0
--local_ip_for_outbound_sockets=127.25.185.195
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36859
--builtin_ntp_servers=127.25.185.212:37103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250809 02:07:12.477828 31230 heartbeater.cc:344] Connected to a master server at 127.25.185.254:36859
I20250809 02:07:12.478324 31230 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:12.479389 31230 heartbeater.cc:507] Master 127.25.185.254:36859 requested a full tablet report, sending...
I20250809 02:07:12.481604 30909 ts_manager.cc:194] Registered new tserver with Master: 25cde50f92a443ae86471232863f6a3b (127.25.185.194:40035)
I20250809 02:07:12.483400 30909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.194:45111
W20250809 02:07:12.771940 31234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:12.772517 31234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:12.773057 31234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:12.803725 31234 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 02:07:12.804116 31234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:12.804883 31234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.195
I20250809 02:07:12.838771 31234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:37103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36859
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.195
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:12.840095 31234 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:12.841652 31234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:12.853011 31240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:13.487028 31230 heartbeater.cc:499] Master 127.25.185.254:36859 was elected leader, sending a full tablet report...
W20250809 02:07:12.854817 31241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:14.047447 31243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:14.049923 31242 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1190 milliseconds
I20250809 02:07:14.050024 31234 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:14.051714 31234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:14.054803 31234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:14.056319 31234 hybrid_clock.cc:648] HybridClock initialized: now 1754705234056239 us; error 77 us; skew 500 ppm
I20250809 02:07:14.057615 31234 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:14.065937 31234 webserver.cc:489] Webserver started at http://127.25.185.195:37443/ using document root <none> and password file <none>
I20250809 02:07:14.067234 31234 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:14.067566 31234 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:14.068205 31234 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:14.075853 31234 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "f5e28e50ff034676abecad60d11c6969"
format_stamp: "Formatted at 2025-08-09 02:07:14 on dist-test-slave-24ff"
I20250809 02:07:14.077471 31234 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "f5e28e50ff034676abecad60d11c6969"
format_stamp: "Formatted at 2025-08-09 02:07:14 on dist-test-slave-24ff"
I20250809 02:07:14.087565 31234 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.004s
I20250809 02:07:14.099613 31250 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:14.100912 31234 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.002s sys 0.005s
I20250809 02:07:14.101275 31234 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "f5e28e50ff034676abecad60d11c6969"
format_stamp: "Formatted at 2025-08-09 02:07:14 on dist-test-slave-24ff"
I20250809 02:07:14.101680 31234 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:14.152258 31234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:14.153697 31234 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:14.154157 31234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:14.156984 31234 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:14.161255 31234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:07:14.161459 31234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:14.161700 31234 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:07:14.161885 31234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:14.294903 31234 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.195:34183
I20250809 02:07:14.295001 31362 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.195:34183 every 8 connection(s)
I20250809 02:07:14.297405 31234 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250809 02:07:14.299114 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 31234
I20250809 02:07:14.299698 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250809 02:07:14.306778 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.196:0
--local_ip_for_outbound_sockets=127.25.185.196
--webserver_interface=127.25.185.196
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36859
--builtin_ntp_servers=127.25.185.212:37103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250809 02:07:14.325079 31363 heartbeater.cc:344] Connected to a master server at 127.25.185.254:36859
I20250809 02:07:14.325610 31363 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:14.326854 31363 heartbeater.cc:507] Master 127.25.185.254:36859 requested a full tablet report, sending...
I20250809 02:07:14.329056 30909 ts_manager.cc:194] Registered new tserver with Master: f5e28e50ff034676abecad60d11c6969 (127.25.185.195:34183)
I20250809 02:07:14.330209 30909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.195:58091
W20250809 02:07:14.602550 31367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:14.603107 31367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:14.603613 31367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:14.634003 31367 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 02:07:14.634433 31367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:14.635277 31367 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.196
I20250809 02:07:14.669822 31367 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:37103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.25.185.196
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36859
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.196
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:14.671280 31367 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:14.672907 31367 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:14.684290 31373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:15.333202 31363 heartbeater.cc:499] Master 127.25.185.254:36859 was elected leader, sending a full tablet report...
W20250809 02:07:16.088492 31372 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 31367
W20250809 02:07:16.424993 31367 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.740s user 0.647s sys 1.091s
W20250809 02:07:16.425405 31375 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1740 milliseconds
W20250809 02:07:14.684803 31374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:16.425420 31367 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.741s user 0.647s sys 1.091s
W20250809 02:07:16.426383 31376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:16.426518 31367 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:16.432015 31367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:16.434221 31367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:16.435580 31367 hybrid_clock.cc:648] HybridClock initialized: now 1754705236435540 us; error 53 us; skew 500 ppm
I20250809 02:07:16.436357 31367 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:16.442307 31367 webserver.cc:489] Webserver started at http://127.25.185.196:33003/ using document root <none> and password file <none>
I20250809 02:07:16.443315 31367 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:16.443555 31367 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:16.444005 31367 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:16.448381 31367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "64ff6f60c2fe4cf9b3f6660d6c2160d1"
format_stamp: "Formatted at 2025-08-09 02:07:16 on dist-test-slave-24ff"
I20250809 02:07:16.449486 31367 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "64ff6f60c2fe4cf9b3f6660d6c2160d1"
format_stamp: "Formatted at 2025-08-09 02:07:16 on dist-test-slave-24ff"
I20250809 02:07:16.457185 31367 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250809 02:07:16.463148 31384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:16.464298 31367 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250809 02:07:16.464645 31367 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "64ff6f60c2fe4cf9b3f6660d6c2160d1"
format_stamp: "Formatted at 2025-08-09 02:07:16 on dist-test-slave-24ff"
I20250809 02:07:16.464988 31367 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:16.511761 31367 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:16.513182 31367 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:16.513651 31367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:16.516136 31367 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:16.520095 31367 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:07:16.520314 31367 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:16.520563 31367 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:07:16.520725 31367 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:16.654230 31367 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.196:38973
I20250809 02:07:16.654399 31496 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.196:38973 every 8 connection(s)
I20250809 02:07:16.656927 31367 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250809 02:07:16.665351 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 31367
I20250809 02:07:16.665781 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250809 02:07:16.672736 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.197:0
--local_ip_for_outbound_sockets=127.25.185.197
--webserver_interface=127.25.185.197
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36859
--builtin_ntp_servers=127.25.185.212:37103
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250809 02:07:16.680383 31497 heartbeater.cc:344] Connected to a master server at 127.25.185.254:36859
I20250809 02:07:16.680912 31497 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:16.682261 31497 heartbeater.cc:507] Master 127.25.185.254:36859 requested a full tablet report, sending...
I20250809 02:07:16.684798 30909 ts_manager.cc:194] Registered new tserver with Master: 64ff6f60c2fe4cf9b3f6660d6c2160d1 (127.25.185.196:38973)
I20250809 02:07:16.686144 30909 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.196:44447
W20250809 02:07:16.974046 31501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:16.974558 31501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:16.975088 31501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:17.005767 31501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250809 02:07:17.006193 31501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:17.007006 31501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.197
I20250809 02:07:17.041386 31501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:37103
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.25.185.197
--webserver_port=0
--tserver_master_addrs=127.25.185.254:36859
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.197
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:17.042804 31501 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:17.044340 31501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:17.055846 31507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:17.690038 31497 heartbeater.cc:499] Master 127.25.185.254:36859 was elected leader, sending a full tablet report...
W20250809 02:07:17.056627 31508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:17.060806 31510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:18.236963 31509 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250809 02:07:18.237047 31501 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:18.240338 31501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:18.242537 31501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:18.243945 31501 hybrid_clock.cc:648] HybridClock initialized: now 1754705238243891 us; error 74 us; skew 500 ppm
I20250809 02:07:18.244702 31501 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:18.250540 31501 webserver.cc:489] Webserver started at http://127.25.185.197:41373/ using document root <none> and password file <none>
I20250809 02:07:18.251461 31501 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:18.251663 31501 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:18.252074 31501 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:18.256477 31501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "b386e8b6eeab4194baafe949a03c11e5"
format_stamp: "Formatted at 2025-08-09 02:07:18 on dist-test-slave-24ff"
I20250809 02:07:18.257586 31501 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "b386e8b6eeab4194baafe949a03c11e5"
format_stamp: "Formatted at 2025-08-09 02:07:18 on dist-test-slave-24ff"
I20250809 02:07:18.265022 31501 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250809 02:07:18.270759 31518 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:18.271852 31501 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250809 02:07:18.272171 31501 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "b386e8b6eeab4194baafe949a03c11e5"
format_stamp: "Formatted at 2025-08-09 02:07:18 on dist-test-slave-24ff"
I20250809 02:07:18.272549 31501 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:18.322454 31501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:18.323880 31501 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:18.324311 31501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:18.326798 31501 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:18.330770 31501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:07:18.330981 31501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:18.331221 31501 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:07:18.331382 31501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:18.490343 31501 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.197:41847
I20250809 02:07:18.490483 31630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.197:41847 every 8 connection(s)
I20250809 02:07:18.492935 31501 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250809 02:07:18.502509 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 31501
I20250809 02:07:18.503427 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250809 02:07:18.516904 31631 heartbeater.cc:344] Connected to a master server at 127.25.185.254:36859
I20250809 02:07:18.517304 31631 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:18.518745 31631 heartbeater.cc:507] Master 127.25.185.254:36859 requested a full tablet report, sending...
I20250809 02:07:18.520781 30908 ts_manager.cc:194] Registered new tserver with Master: b386e8b6eeab4194baafe949a03c11e5 (127.25.185.197:41847)
I20250809 02:07:18.522192 30908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.197:35073
I20250809 02:07:18.525691 26343 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250809 02:07:18.563027 30909 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46978:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250809 02:07:18.651494 31165 tablet_service.cc:1468] Processing CreateTablet for tablet 6e2a08d9b20f467da4889ed2d5868a80 (DEFAULT_TABLE table=TestTable [id=ce7b5a5cb2084473b622833bb98bc2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:07:18.653769 31165 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2a08d9b20f467da4889ed2d5868a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:18.664170 31298 tablet_service.cc:1468] Processing CreateTablet for tablet 6e2a08d9b20f467da4889ed2d5868a80 (DEFAULT_TABLE table=TestTable [id=ce7b5a5cb2084473b622833bb98bc2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:07:18.664687 31032 tablet_service.cc:1468] Processing CreateTablet for tablet 6e2a08d9b20f467da4889ed2d5868a80 (DEFAULT_TABLE table=TestTable [id=ce7b5a5cb2084473b622833bb98bc2f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:07:18.666018 31298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2a08d9b20f467da4889ed2d5868a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:18.666352 31032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e2a08d9b20f467da4889ed2d5868a80. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:18.693714 31650 tablet_bootstrap.cc:492] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b: Bootstrap starting.
I20250809 02:07:18.698638 31652 tablet_bootstrap.cc:492] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110: Bootstrap starting.
I20250809 02:07:18.700945 31651 tablet_bootstrap.cc:492] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969: Bootstrap starting.
I20250809 02:07:18.703516 31650 tablet_bootstrap.cc:654] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:18.706148 31650 log.cc:826] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:18.707976 31652 tablet_bootstrap.cc:654] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:18.709403 31651 tablet_bootstrap.cc:654] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:18.711172 31652 log.cc:826] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:18.711689 31651 log.cc:826] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:18.712231 31650 tablet_bootstrap.cc:492] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b: No bootstrap required, opened a new log
I20250809 02:07:18.712636 31650 ts_tablet_manager.cc:1397] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b: Time spent bootstrapping tablet: real 0.020s user 0.016s sys 0.000s
I20250809 02:07:18.718334 31652 tablet_bootstrap.cc:492] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110: No bootstrap required, opened a new log
I20250809 02:07:18.718485 31651 tablet_bootstrap.cc:492] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969: No bootstrap required, opened a new log
I20250809 02:07:18.719014 31652 ts_tablet_manager.cc:1397] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110: Time spent bootstrapping tablet: real 0.021s user 0.016s sys 0.003s
I20250809 02:07:18.719069 31651 ts_tablet_manager.cc:1397] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.013s
I20250809 02:07:18.740002 31650 raft_consensus.cc:357] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 } } peers { permanent_uuid: "590350e6d4044505a84578f3c330b110" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40993 } } peers { permanent_uuid: "f5e28e50ff034676abecad60d11c6969" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34183 } }
I20250809 02:07:18.741035 31650 raft_consensus.cc:383] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:18.741339 31650 raft_consensus.cc:738] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25cde50f92a443ae86471232863f6a3b, State: Initialized, Role: FOLLOWER
I20250809 02:07:18.741837 31652 raft_consensus.cc:357] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 } } peers { permanent_uuid: "590350e6d4044505a84578f3c330b110" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40993 } } peers { permanent_uuid: "f5e28e50ff034676abecad60d11c6969" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34183 } }
I20250809 02:07:18.742717 31652 raft_consensus.cc:383] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:18.742971 31652 raft_consensus.cc:738] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 590350e6d4044505a84578f3c330b110, State: Initialized, Role: FOLLOWER
I20250809 02:07:18.742939 31650 consensus_queue.cc:260] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 } } peers { permanent_uuid: "590350e6d4044505a84578f3c330b110" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40993 } } peers { permanent_uuid: "f5e28e50ff034676abecad60d11c6969" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34183 } }
I20250809 02:07:18.743778 31652 consensus_queue.cc:260] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 } } peers { permanent_uuid: "590350e6d4044505a84578f3c330b110" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40993 } } peers { permanent_uuid: "f5e28e50ff034676abecad60d11c6969" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34183 } }
I20250809 02:07:18.747188 31650 ts_tablet_manager.cc:1428] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b: Time spent starting tablet: real 0.034s user 0.036s sys 0.000s
I20250809 02:07:18.747855 31651 raft_consensus.cc:357] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 } } peers { permanent_uuid: "590350e6d4044505a84578f3c330b110" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40993 } } peers { permanent_uuid: "f5e28e50ff034676abecad60d11c6969" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34183 } }
I20250809 02:07:18.748876 31651 raft_consensus.cc:383] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:18.749240 31651 raft_consensus.cc:738] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5e28e50ff034676abecad60d11c6969, State: Initialized, Role: FOLLOWER
I20250809 02:07:18.750301 31652 ts_tablet_manager.cc:1428] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110: Time spent starting tablet: real 0.031s user 0.013s sys 0.016s
I20250809 02:07:18.750196 31651 consensus_queue.cc:260] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 } } peers { permanent_uuid: "590350e6d4044505a84578f3c330b110" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40993 } } peers { permanent_uuid: "f5e28e50ff034676abecad60d11c6969" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34183 } }
I20250809 02:07:18.756564 31651 ts_tablet_manager.cc:1428] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969: Time spent starting tablet: real 0.037s user 0.025s sys 0.011s
I20250809 02:07:18.803778 31657 raft_consensus.cc:491] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:07:18.804280 31657 raft_consensus.cc:513] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 } } peers { permanent_uuid: "590350e6d4044505a84578f3c330b110" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40993 } } peers { permanent_uuid: "f5e28e50ff034676abecad60d11c6969" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34183 } }
W20250809 02:07:18.808183 31364 tablet.cc:2378] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:07:18.807076 31657 leader_election.cc:290] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 25cde50f92a443ae86471232863f6a3b (127.25.185.194:40035), f5e28e50ff034676abecad60d11c6969 (127.25.185.195:34183)
I20250809 02:07:18.829667 31318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e2a08d9b20f467da4889ed2d5868a80" candidate_uuid: "590350e6d4044505a84578f3c330b110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e28e50ff034676abecad60d11c6969" is_pre_election: true
I20250809 02:07:18.829663 31185 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e2a08d9b20f467da4889ed2d5868a80" candidate_uuid: "590350e6d4044505a84578f3c330b110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cde50f92a443ae86471232863f6a3b" is_pre_election: true
I20250809 02:07:18.830359 31318 raft_consensus.cc:2466] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590350e6d4044505a84578f3c330b110 in term 0.
I20250809 02:07:18.830403 31185 raft_consensus.cc:2466] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 590350e6d4044505a84578f3c330b110 in term 0.
I20250809 02:07:18.831772 30987 leader_election.cc:304] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cde50f92a443ae86471232863f6a3b, 590350e6d4044505a84578f3c330b110; no voters:
I20250809 02:07:18.832981 31657 raft_consensus.cc:2802] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 02:07:18.833515 31657 raft_consensus.cc:491] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 02:07:18.833884 31657 raft_consensus.cc:3058] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:18.839566 31657 raft_consensus.cc:513] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 } } peers { permanent_uuid: "590350e6d4044505a84578f3c330b110" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40993 } } peers { permanent_uuid: "f5e28e50ff034676abecad60d11c6969" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34183 } }
I20250809 02:07:18.841250 31657 leader_election.cc:290] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [CANDIDATE]: Term 1 election: Requested vote from peers 25cde50f92a443ae86471232863f6a3b (127.25.185.194:40035), f5e28e50ff034676abecad60d11c6969 (127.25.185.195:34183)
I20250809 02:07:18.842154 31185 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e2a08d9b20f467da4889ed2d5868a80" candidate_uuid: "590350e6d4044505a84578f3c330b110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "25cde50f92a443ae86471232863f6a3b"
I20250809 02:07:18.842305 31318 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e2a08d9b20f467da4889ed2d5868a80" candidate_uuid: "590350e6d4044505a84578f3c330b110" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5e28e50ff034676abecad60d11c6969"
I20250809 02:07:18.842728 31185 raft_consensus.cc:3058] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:18.842885 31318 raft_consensus.cc:3058] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:18.849830 31318 raft_consensus.cc:2466] T 6e2a08d9b20f467da4889ed2d5868a80 P f5e28e50ff034676abecad60d11c6969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590350e6d4044505a84578f3c330b110 in term 1.
I20250809 02:07:18.849830 31185 raft_consensus.cc:2466] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 590350e6d4044505a84578f3c330b110 in term 1.
I20250809 02:07:18.850920 30987 leader_election.cc:304] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25cde50f92a443ae86471232863f6a3b, 590350e6d4044505a84578f3c330b110; no voters:
I20250809 02:07:18.851572 31657 raft_consensus.cc:2802] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:07:18.853158 31657 raft_consensus.cc:695] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [term 1 LEADER]: Becoming Leader. State: Replica: 590350e6d4044505a84578f3c330b110, State: Running, Role: LEADER
I20250809 02:07:18.854354 31657 consensus_queue.cc:237] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 } } peers { permanent_uuid: "590350e6d4044505a84578f3c330b110" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40993 } } peers { permanent_uuid: "f5e28e50ff034676abecad60d11c6969" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34183 } }
I20250809 02:07:18.863889 30906 catalog_manager.cc:5582] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 reported cstate change: term changed from 0 to 1, leader changed from <none> to 590350e6d4044505a84578f3c330b110 (127.25.185.193). New cstate: current_term: 1 leader_uuid: "590350e6d4044505a84578f3c330b110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "590350e6d4044505a84578f3c330b110" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5e28e50ff034676abecad60d11c6969" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 34183 } health_report { overall_health: UNKNOWN } } }
I20250809 02:07:18.911010 26343 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250809 02:07:18.914232 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 590350e6d4044505a84578f3c330b110 to finish bootstrapping
W20250809 02:07:18.926556 31098 tablet.cc:2378] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:07:18.928247 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 25cde50f92a443ae86471232863f6a3b to finish bootstrapping
I20250809 02:07:18.938555 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f5e28e50ff034676abecad60d11c6969 to finish bootstrapping
I20250809 02:07:18.949939 26343 test_util.cc:276] Using random seed: 645136412
W20250809 02:07:18.968945 31231 tablet.cc:2378] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:07:18.975025 26343 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250809 02:07:18.975884 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 31234
W20250809 02:07:19.008451 30985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.185.195:34183: connect: Connection refused (error 111)
I20250809 02:07:19.009927 31185 raft_consensus.cc:1273] T 6e2a08d9b20f467da4889ed2d5868a80 P 25cde50f92a443ae86471232863f6a3b [term 1 FOLLOWER]: Refusing update from remote peer 590350e6d4044505a84578f3c330b110: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250809 02:07:19.012048 31657 consensus_queue.cc:1035] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "25cde50f92a443ae86471232863f6a3b" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 40035 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250809 02:07:19.014467 30985 consensus_peers.cc:489] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 -> Peer f5e28e50ff034676abecad60d11c6969 (127.25.185.195:34183): Couldn't send request to peer f5e28e50ff034676abecad60d11c6969. Status: Network error: Client connection negotiation failed: client connection to 127.25.185.195:34183: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250809 02:07:19.031201 31672 mvcc.cc:204] Tried to move back new op lower bound from 7187272658962296832 to 7187272658356191232. Current Snapshot: MvccSnapshot[applied={T|T < 7187272658962296832}]
I20250809 02:07:19.525298 31631 heartbeater.cc:499] Master 127.25.185.254:36859 was elected leader, sending a full tablet report...
W20250809 02:07:20.011237 30985 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.25.185.195:34183, user_credentials={real_user=slave}} blocked reactor thread for 73382.3us
W20250809 02:07:21.520509 30985 consensus_peers.cc:489] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 -> Peer f5e28e50ff034676abecad60d11c6969 (127.25.185.195:34183): Couldn't send request to peer f5e28e50ff034676abecad60d11c6969. Status: Network error: Client connection negotiation failed: client connection to 127.25.185.195:34183: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250809 02:07:21.826627 31165 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 02:07:21.844614 31032 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 02:07:21.873593 31432 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 02:07:21.896309 31566 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 02:07:23.671782 31165 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 02:07:23.684276 31432 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 02:07:23.687052 31566 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 02:07:23.703141 31032 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
W20250809 02:07:23.943549 30985 consensus_peers.cc:489] T 6e2a08d9b20f467da4889ed2d5868a80 P 590350e6d4044505a84578f3c330b110 -> Peer f5e28e50ff034676abecad60d11c6969 (127.25.185.195:34183): Couldn't send request to peer f5e28e50ff034676abecad60d11c6969. Status: Network error: Client connection negotiation failed: client connection to 127.25.185.195:34183: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250809 02:07:24.500608 30985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.25.185.195:34183: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250809 02:07:26.132122 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 30968
I20250809 02:07:26.172946 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 31101
I20250809 02:07:26.206337 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 31367
I20250809 02:07:26.231024 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 31501
I20250809 02:07:26.257809 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 30875
2025-08-09T02:07:26Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (20346 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (20346 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250809 02:07:26.317200 26343 test_util.cc:276] Using random seed: 652503671
I20250809 02:07:26.321115 26343 ts_itest-base.cc:115] Starting cluster with:
I20250809 02:07:26.321264 26343 ts_itest-base.cc:116] --------------
I20250809 02:07:26.321377 26343 ts_itest-base.cc:117] 1 tablet servers
I20250809 02:07:26.321501 26343 ts_itest-base.cc:118] 1 replicas per TS
I20250809 02:07:26.321609 26343 ts_itest-base.cc:119] --------------
2025-08-09T02:07:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T02:07:26Z Disabled control of system clock
I20250809 02:07:26.366233 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:46349
--webserver_interface=127.25.185.254
--webserver_port=0
--builtin_ntp_servers=127.25.185.212:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:46349 with env {}
W20250809 02:07:26.667865 31771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:26.668426 31771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:26.668824 31771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:26.700145 31771 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:07:26.700457 31771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:26.700726 31771 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:07:26.700963 31771 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:07:26.736313 31771 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:46349
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:46349
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:26.737534 31771 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:26.739388 31771 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:26.749974 31777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:26.751315 31778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:28.152626 31780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:28.154338 31771 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.404s user 0.524s sys 0.876s
W20250809 02:07:28.154778 31771 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.405s user 0.526s sys 0.878s
W20250809 02:07:28.156505 31779 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1400 milliseconds
I20250809 02:07:28.156566 31771 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:28.157871 31771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:28.160526 31771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:28.161875 31771 hybrid_clock.cc:648] HybridClock initialized: now 1754705248161846 us; error 41 us; skew 500 ppm
I20250809 02:07:28.162699 31771 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:28.169873 31771 webserver.cc:489] Webserver started at http://127.25.185.254:44107/ using document root <none> and password file <none>
I20250809 02:07:28.170818 31771 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:28.171047 31771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:28.171501 31771 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:28.175899 31771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "09db7fad5a004719b3eba05f451f610e"
format_stamp: "Formatted at 2025-08-09 02:07:28 on dist-test-slave-24ff"
I20250809 02:07:28.176982 31771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "09db7fad5a004719b3eba05f451f610e"
format_stamp: "Formatted at 2025-08-09 02:07:28 on dist-test-slave-24ff"
I20250809 02:07:28.184805 31771 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250809 02:07:28.190867 31787 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:28.192234 31771 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250809 02:07:28.192585 31771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
uuid: "09db7fad5a004719b3eba05f451f610e"
format_stamp: "Formatted at 2025-08-09 02:07:28 on dist-test-slave-24ff"
I20250809 02:07:28.192922 31771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:28.266037 31771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:28.267604 31771 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:28.268056 31771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:28.339331 31771 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:46349
I20250809 02:07:28.339397 31839 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:46349 every 8 connection(s)
I20250809 02:07:28.342088 31771 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250809 02:07:28.346261 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 31771
I20250809 02:07:28.346979 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250809 02:07:28.347970 31840 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:28.372486 31840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e: Bootstrap starting.
I20250809 02:07:28.377825 31840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:28.379869 31840 log.cc:826] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:28.384379 31840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e: No bootstrap required, opened a new log
I20250809 02:07:28.401830 31840 raft_consensus.cc:357] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db7fad5a004719b3eba05f451f610e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46349 } }
I20250809 02:07:28.402464 31840 raft_consensus.cc:383] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:28.402745 31840 raft_consensus.cc:738] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09db7fad5a004719b3eba05f451f610e, State: Initialized, Role: FOLLOWER
I20250809 02:07:28.403416 31840 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db7fad5a004719b3eba05f451f610e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46349 } }
I20250809 02:07:28.403944 31840 raft_consensus.cc:397] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:07:28.404225 31840 raft_consensus.cc:491] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:07:28.404523 31840 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:28.408962 31840 raft_consensus.cc:513] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db7fad5a004719b3eba05f451f610e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46349 } }
I20250809 02:07:28.409641 31840 leader_election.cc:304] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09db7fad5a004719b3eba05f451f610e; no voters:
I20250809 02:07:28.411626 31840 leader_election.cc:290] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:07:28.412473 31845 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:07:28.414701 31845 raft_consensus.cc:695] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [term 1 LEADER]: Becoming Leader. State: Replica: 09db7fad5a004719b3eba05f451f610e, State: Running, Role: LEADER
I20250809 02:07:28.415446 31845 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db7fad5a004719b3eba05f451f610e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46349 } }
I20250809 02:07:28.416401 31840 sys_catalog.cc:564] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:07:28.423477 31847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09db7fad5a004719b3eba05f451f610e. Latest consensus state: current_term: 1 leader_uuid: "09db7fad5a004719b3eba05f451f610e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db7fad5a004719b3eba05f451f610e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46349 } } }
I20250809 02:07:28.423369 31846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09db7fad5a004719b3eba05f451f610e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09db7fad5a004719b3eba05f451f610e" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46349 } } }
I20250809 02:07:28.424093 31847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [sys.catalog]: This master's current role is: LEADER
I20250809 02:07:28.424264 31846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e [sys.catalog]: This master's current role is: LEADER
I20250809 02:07:28.432582 31853 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:07:28.443239 31853 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:07:28.460157 31853 catalog_manager.cc:1349] Generated new cluster ID: 375a35e6c5e54ababdda28b1bd948cf9
I20250809 02:07:28.460476 31853 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:07:28.485469 31853 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 02:07:28.486990 31853 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:07:28.503257 31853 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 09db7fad5a004719b3eba05f451f610e: Generated new TSK 0
I20250809 02:07:28.504256 31853 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 02:07:28.528137 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:0
--local_ip_for_outbound_sockets=127.25.185.193
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:46349
--builtin_ntp_servers=127.25.185.212:42571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250809 02:07:28.831037 31864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:28.831566 31864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:28.832062 31864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:28.863034 31864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:28.863885 31864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:07:28.898900 31864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:42571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:46349
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:28.900241 31864 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:28.901867 31864 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:28.914533 31870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:28.915504 31871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:30.317343 31869 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 31864
W20250809 02:07:30.352411 31864 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.436s user 0.002s sys 0.000s
W20250809 02:07:30.352715 31864 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.436s user 0.002s sys 0.000s
W20250809 02:07:30.364291 31873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:30.365223 31864 server_base.cc:1047] running on GCE node
I20250809 02:07:30.366323 31864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:30.368891 31864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:30.370364 31864 hybrid_clock.cc:648] HybridClock initialized: now 1754705250370302 us; error 57 us; skew 500 ppm
I20250809 02:07:30.371201 31864 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:30.378050 31864 webserver.cc:489] Webserver started at http://127.25.185.193:39511/ using document root <none> and password file <none>
I20250809 02:07:30.379032 31864 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:30.379251 31864 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:30.379721 31864 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:30.383920 31864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e50fef8b8eb844f99ea383ebec3d50db"
format_stamp: "Formatted at 2025-08-09 02:07:30 on dist-test-slave-24ff"
I20250809 02:07:30.384994 31864 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e50fef8b8eb844f99ea383ebec3d50db"
format_stamp: "Formatted at 2025-08-09 02:07:30 on dist-test-slave-24ff"
I20250809 02:07:30.392241 31864 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.000s sys 0.008s
I20250809 02:07:30.398106 31880 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:30.399209 31864 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250809 02:07:30.399505 31864 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e50fef8b8eb844f99ea383ebec3d50db"
format_stamp: "Formatted at 2025-08-09 02:07:30 on dist-test-slave-24ff"
I20250809 02:07:30.399832 31864 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:30.451170 31864 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:30.452551 31864 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:30.452960 31864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:30.455998 31864 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:30.459987 31864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:07:30.460186 31864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:30.460496 31864 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:07:30.460660 31864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:30.627187 31864 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:40113
I20250809 02:07:30.627308 31992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:40113 every 8 connection(s)
I20250809 02:07:30.629786 31864 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250809 02:07:30.633255 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 31864
I20250809 02:07:30.633739 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754705097773638-26343-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250809 02:07:30.662870 31993 heartbeater.cc:344] Connected to a master server at 127.25.185.254:46349
I20250809 02:07:30.663391 31993 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:30.664672 31993 heartbeater.cc:507] Master 127.25.185.254:46349 requested a full tablet report, sending...
I20250809 02:07:30.667480 31805 ts_manager.cc:194] Registered new tserver with Master: e50fef8b8eb844f99ea383ebec3d50db (127.25.185.193:40113)
I20250809 02:07:30.669329 31805 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:54271
I20250809 02:07:30.675177 26343 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250809 02:07:30.710465 31805 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45222:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250809 02:07:30.771456 31928 tablet_service.cc:1468] Processing CreateTablet for tablet 3aa6ac7e06a3477a80f029119df9c042 (DEFAULT_TABLE table=TestTable [id=b06dd624000348efbcf71c5316c24a9d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:07:30.772902 31928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3aa6ac7e06a3477a80f029119df9c042. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:30.792330 32008 tablet_bootstrap.cc:492] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db: Bootstrap starting.
I20250809 02:07:30.798303 32008 tablet_bootstrap.cc:654] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:30.800055 32008 log.cc:826] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:30.804988 32008 tablet_bootstrap.cc:492] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db: No bootstrap required, opened a new log
I20250809 02:07:30.805397 32008 ts_tablet_manager.cc:1397] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.007s
I20250809 02:07:30.822847 32008 raft_consensus.cc:357] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50fef8b8eb844f99ea383ebec3d50db" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40113 } }
I20250809 02:07:30.823616 32008 raft_consensus.cc:383] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:30.823866 32008 raft_consensus.cc:738] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e50fef8b8eb844f99ea383ebec3d50db, State: Initialized, Role: FOLLOWER
I20250809 02:07:30.824571 32008 consensus_queue.cc:260] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50fef8b8eb844f99ea383ebec3d50db" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40113 } }
I20250809 02:07:30.825089 32008 raft_consensus.cc:397] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:07:30.825371 32008 raft_consensus.cc:491] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:07:30.825760 32008 raft_consensus.cc:3058] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:30.830672 32008 raft_consensus.cc:513] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50fef8b8eb844f99ea383ebec3d50db" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40113 } }
I20250809 02:07:30.831377 32008 leader_election.cc:304] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e50fef8b8eb844f99ea383ebec3d50db; no voters:
I20250809 02:07:30.833666 32008 leader_election.cc:290] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:07:30.834591 32010 raft_consensus.cc:2802] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:07:30.837306 31993 heartbeater.cc:499] Master 127.25.185.254:46349 was elected leader, sending a full tablet report...
I20250809 02:07:30.837847 32008 ts_tablet_manager.cc:1428] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db: Time spent starting tablet: real 0.032s user 0.026s sys 0.006s
I20250809 02:07:30.838541 32010 raft_consensus.cc:695] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [term 1 LEADER]: Becoming Leader. State: Replica: e50fef8b8eb844f99ea383ebec3d50db, State: Running, Role: LEADER
I20250809 02:07:30.839236 32010 consensus_queue.cc:237] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50fef8b8eb844f99ea383ebec3d50db" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40113 } }
I20250809 02:07:30.851894 31805 catalog_manager.cc:5582] T 3aa6ac7e06a3477a80f029119df9c042 P e50fef8b8eb844f99ea383ebec3d50db reported cstate change: term changed from 0 to 1, leader changed from <none> to e50fef8b8eb844f99ea383ebec3d50db (127.25.185.193). New cstate: current_term: 1 leader_uuid: "e50fef8b8eb844f99ea383ebec3d50db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e50fef8b8eb844f99ea383ebec3d50db" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 40113 } health_report { overall_health: HEALTHY } } }
I20250809 02:07:30.876668 26343 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250809 02:07:30.880862 26343 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e50fef8b8eb844f99ea383ebec3d50db to finish bootstrapping
I20250809 02:07:33.589452 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 31864
I20250809 02:07:33.614485 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 31771
2025-08-09T02:07:33Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7353 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7353 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250809 02:07:33.670986 26343 test_util.cc:276] Using random seed: 659857455
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-09T02:07:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-09T02:07:33Z Disabled control of system clock
I20250809 02:07:33.722801 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:46623
--webserver_interface=127.25.185.254
--webserver_port=0
--builtin_ntp_servers=127.25.185.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:46623 with env {}
W20250809 02:07:34.025439 32036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:34.026057 32036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:34.026520 32036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:34.057616 32036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:07:34.057934 32036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:34.058235 32036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:07:34.058476 32036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:07:34.093519 32036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:46623
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:34.094918 32036 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:34.096585 32036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:34.107652 32042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:34.108320 32043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:35.312947 32045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:35.316082 32044 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1207 milliseconds
I20250809 02:07:35.316193 32036 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:35.317431 32036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:35.320529 32036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:35.321998 32036 hybrid_clock.cc:648] HybridClock initialized: now 1754705255321964 us; error 40 us; skew 500 ppm
I20250809 02:07:35.322858 32036 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:35.329432 32036 webserver.cc:489] Webserver started at http://127.25.185.254:45833/ using document root <none> and password file <none>
I20250809 02:07:35.330330 32036 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:35.330526 32036 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:35.330991 32036 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:35.335423 32036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/instance:
uuid: "d5ec8b5582834deb8012ff2f298c4040"
format_stamp: "Formatted at 2025-08-09 02:07:35 on dist-test-slave-24ff"
I20250809 02:07:35.336498 32036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal/instance:
uuid: "d5ec8b5582834deb8012ff2f298c4040"
format_stamp: "Formatted at 2025-08-09 02:07:35 on dist-test-slave-24ff"
I20250809 02:07:35.343688 32036 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250809 02:07:35.349117 32052 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:35.350112 32036 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250809 02:07:35.350399 32036 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
uuid: "d5ec8b5582834deb8012ff2f298c4040"
format_stamp: "Formatted at 2025-08-09 02:07:35 on dist-test-slave-24ff"
I20250809 02:07:35.350718 32036 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:35.413713 32036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:35.415222 32036 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:35.415627 32036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:35.485380 32036 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:46623
I20250809 02:07:35.485441 32103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:46623 every 8 connection(s)
I20250809 02:07:35.488176 32036 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/info.pb
I20250809 02:07:35.493111 32104 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:35.495404 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 32036
I20250809 02:07:35.495823 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal/instance
I20250809 02:07:35.515767 32104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040: Bootstrap starting.
I20250809 02:07:35.520748 32104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:35.522377 32104 log.cc:826] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:35.527420 32104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040: No bootstrap required, opened a new log
I20250809 02:07:35.543526 32104 raft_consensus.cc:357] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ec8b5582834deb8012ff2f298c4040" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } }
I20250809 02:07:35.544189 32104 raft_consensus.cc:383] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:35.544421 32104 raft_consensus.cc:738] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ec8b5582834deb8012ff2f298c4040, State: Initialized, Role: FOLLOWER
I20250809 02:07:35.545078 32104 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ec8b5582834deb8012ff2f298c4040" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } }
I20250809 02:07:35.545570 32104 raft_consensus.cc:397] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:07:35.545825 32104 raft_consensus.cc:491] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:07:35.546155 32104 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:35.550139 32104 raft_consensus.cc:513] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ec8b5582834deb8012ff2f298c4040" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } }
I20250809 02:07:35.550835 32104 leader_election.cc:304] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5ec8b5582834deb8012ff2f298c4040; no voters:
I20250809 02:07:35.552450 32104 leader_election.cc:290] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:07:35.553334 32109 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:07:35.555723 32109 raft_consensus.cc:695] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [term 1 LEADER]: Becoming Leader. State: Replica: d5ec8b5582834deb8012ff2f298c4040, State: Running, Role: LEADER
I20250809 02:07:35.556679 32104 sys_catalog.cc:564] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:07:35.556484 32109 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ec8b5582834deb8012ff2f298c4040" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } }
I20250809 02:07:35.567178 32110 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d5ec8b5582834deb8012ff2f298c4040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ec8b5582834deb8012ff2f298c4040" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } } }
I20250809 02:07:35.568188 32110 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [sys.catalog]: This master's current role is: LEADER
I20250809 02:07:35.567809 32111 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ec8b5582834deb8012ff2f298c4040. Latest consensus state: current_term: 1 leader_uuid: "d5ec8b5582834deb8012ff2f298c4040" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5ec8b5582834deb8012ff2f298c4040" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } } }
I20250809 02:07:35.568645 32111 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040 [sys.catalog]: This master's current role is: LEADER
I20250809 02:07:35.572120 32118 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:07:35.586447 32118 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:07:35.603205 32118 catalog_manager.cc:1349] Generated new cluster ID: d26b5268cba84c548d4ef35187721cf4
I20250809 02:07:35.603504 32118 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:07:35.625108 32118 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 02:07:35.627148 32118 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:07:35.643723 32118 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d5ec8b5582834deb8012ff2f298c4040: Generated new TSK 0
I20250809 02:07:35.644798 32118 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250809 02:07:35.667600 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:0
--local_ip_for_outbound_sockets=127.25.185.193
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:46623
--builtin_ntp_servers=127.25.185.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250809 02:07:36.001121 32128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:36.001709 32128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:36.002197 32128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:36.034844 32128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:36.035687 32128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:07:36.071372 32128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=0
--tserver_master_addrs=127.25.185.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:36.072602 32128 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:36.074186 32128 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:36.086723 32134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:37.489996 32133 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 32128
W20250809 02:07:37.537357 32128 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.450s user 0.000s sys 0.002s
W20250809 02:07:36.088064 32135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:37.537752 32128 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.450s user 0.000s sys 0.003s
W20250809 02:07:37.539573 32137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:37.542397 32136 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1448 milliseconds
I20250809 02:07:37.542465 32128 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:37.543912 32128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:37.546459 32128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:37.547971 32128 hybrid_clock.cc:648] HybridClock initialized: now 1754705257547912 us; error 52 us; skew 500 ppm
I20250809 02:07:37.549017 32128 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:37.556275 32128 webserver.cc:489] Webserver started at http://127.25.185.193:36949/ using document root <none> and password file <none>
I20250809 02:07:37.557473 32128 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:37.557739 32128 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:37.558292 32128 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:37.564522 32128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/instance:
uuid: "4fb3097677c84677bf7892c42bf355b6"
format_stamp: "Formatted at 2025-08-09 02:07:37 on dist-test-slave-24ff"
I20250809 02:07:37.566010 32128 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal/instance:
uuid: "4fb3097677c84677bf7892c42bf355b6"
format_stamp: "Formatted at 2025-08-09 02:07:37 on dist-test-slave-24ff"
I20250809 02:07:37.575125 32128 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.005s
I20250809 02:07:37.582419 32144 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:37.583631 32128 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250809 02:07:37.584026 32128 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal
uuid: "4fb3097677c84677bf7892c42bf355b6"
format_stamp: "Formatted at 2025-08-09 02:07:37 on dist-test-slave-24ff"
I20250809 02:07:37.584474 32128 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:37.668892 32128 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:37.670425 32128 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:37.670899 32128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:37.673579 32128 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:37.677963 32128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:07:37.678190 32128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:37.678419 32128 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:07:37.678581 32128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:37.857776 32128 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:36219
I20250809 02:07:37.857779 32256 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:36219 every 8 connection(s)
I20250809 02:07:37.861831 32128 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/info.pb
I20250809 02:07:37.862522 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 32128
I20250809 02:07:37.863080 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal/instance
I20250809 02:07:37.874130 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.194:0
--local_ip_for_outbound_sockets=127.25.185.194
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:46623
--builtin_ntp_servers=127.25.185.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250809 02:07:37.898414 32257 heartbeater.cc:344] Connected to a master server at 127.25.185.254:46623
I20250809 02:07:37.898891 32257 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:37.899900 32257 heartbeater.cc:507] Master 127.25.185.254:46623 requested a full tablet report, sending...
I20250809 02:07:37.902248 32069 ts_manager.cc:194] Registered new tserver with Master: 4fb3097677c84677bf7892c42bf355b6 (127.25.185.193:36219)
I20250809 02:07:37.904232 32069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:34309
W20250809 02:07:38.219810 32261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:38.220371 32261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:38.220889 32261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:38.253252 32261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:38.254096 32261 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.194
I20250809 02:07:38.288714 32261 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.185.194
--webserver_port=0
--tserver_master_addrs=127.25.185.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.194
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:38.290058 32261 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:38.291605 32261 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:38.303789 32268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:38.907459 32257 heartbeater.cc:499] Master 127.25.185.254:46623 was elected leader, sending a full tablet report...
W20250809 02:07:38.308506 32271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:38.304354 32269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:39.450059 32270 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250809 02:07:39.450196 32261 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:39.453833 32261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:39.456562 32261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:39.458004 32261 hybrid_clock.cc:648] HybridClock initialized: now 1754705259457935 us; error 73 us; skew 500 ppm
I20250809 02:07:39.458854 32261 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:39.465509 32261 webserver.cc:489] Webserver started at http://127.25.185.194:44223/ using document root <none> and password file <none>
I20250809 02:07:39.466418 32261 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:39.466604 32261 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:39.467111 32261 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:39.471802 32261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/instance:
uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec"
format_stamp: "Formatted at 2025-08-09 02:07:39 on dist-test-slave-24ff"
I20250809 02:07:39.472971 32261 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal/instance:
uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec"
format_stamp: "Formatted at 2025-08-09 02:07:39 on dist-test-slave-24ff"
I20250809 02:07:39.480165 32261 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250809 02:07:39.486294 32278 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:39.487402 32261 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250809 02:07:39.487702 32261 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal
uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec"
format_stamp: "Formatted at 2025-08-09 02:07:39 on dist-test-slave-24ff"
I20250809 02:07:39.488026 32261 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:39.535362 32261 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:39.536844 32261 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:39.537241 32261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:39.539707 32261 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:39.543606 32261 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:07:39.543860 32261 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:39.544109 32261 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:07:39.544265 32261 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:39.678814 32261 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.194:42421
I20250809 02:07:39.678918 32390 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.194:42421 every 8 connection(s)
I20250809 02:07:39.681315 32261 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/info.pb
I20250809 02:07:39.687498 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 32261
I20250809 02:07:39.688386 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal/instance
I20250809 02:07:39.695546 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.195:0
--local_ip_for_outbound_sockets=127.25.185.195
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:46623
--builtin_ntp_servers=127.25.185.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250809 02:07:39.703881 32391 heartbeater.cc:344] Connected to a master server at 127.25.185.254:46623
I20250809 02:07:39.704309 32391 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:39.705333 32391 heartbeater.cc:507] Master 127.25.185.254:46623 requested a full tablet report, sending...
I20250809 02:07:39.707705 32069 ts_manager.cc:194] Registered new tserver with Master: fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194:42421)
I20250809 02:07:39.709594 32069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.194:34287
W20250809 02:07:40.001588 32395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:40.002069 32395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:40.002579 32395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:40.033722 32395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:40.034730 32395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.195
I20250809 02:07:40.069118 32395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.185.195
--webserver_port=0
--tserver_master_addrs=127.25.185.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.195
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:40.070448 32395 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:40.072335 32395 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:40.084216 32401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:40.713104 32391 heartbeater.cc:499] Master 127.25.185.254:46623 was elected leader, sending a full tablet report...
W20250809 02:07:40.084633 32402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:41.366524 32404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:41.368746 32403 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1279 milliseconds
W20250809 02:07:41.369459 32395 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.286s user 0.408s sys 0.872s
W20250809 02:07:41.369709 32395 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.286s user 0.408s sys 0.872s
I20250809 02:07:41.369928 32395 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:41.370970 32395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:41.373253 32395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:41.374600 32395 hybrid_clock.cc:648] HybridClock initialized: now 1754705261374562 us; error 41 us; skew 500 ppm
I20250809 02:07:41.375459 32395 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:41.382440 32395 webserver.cc:489] Webserver started at http://127.25.185.195:45211/ using document root <none> and password file <none>
I20250809 02:07:41.383474 32395 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:41.383716 32395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:41.384160 32395 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:41.388736 32395 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/instance:
uuid: "fef8948af77f437388b5974a19868a02"
format_stamp: "Formatted at 2025-08-09 02:07:41 on dist-test-slave-24ff"
I20250809 02:07:41.389822 32395 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal/instance:
uuid: "fef8948af77f437388b5974a19868a02"
format_stamp: "Formatted at 2025-08-09 02:07:41 on dist-test-slave-24ff"
I20250809 02:07:41.397475 32395 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250809 02:07:41.403497 32411 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:41.404605 32395 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250809 02:07:41.404938 32395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal
uuid: "fef8948af77f437388b5974a19868a02"
format_stamp: "Formatted at 2025-08-09 02:07:41 on dist-test-slave-24ff"
I20250809 02:07:41.405270 32395 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:41.480929 32395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:41.482393 32395 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:41.482837 32395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:41.485525 32395 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:41.489645 32395 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250809 02:07:41.489838 32395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:41.490083 32395 ts_tablet_manager.cc:610] Registered 0 tablets
I20250809 02:07:41.490221 32395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:41.620136 32395 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.195:43325
I20250809 02:07:41.620235 32523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.195:43325 every 8 connection(s)
I20250809 02:07:41.624212 32395 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/info.pb
I20250809 02:07:41.634876 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 32395
I20250809 02:07:41.635277 26343 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal/instance
I20250809 02:07:41.646946 32524 heartbeater.cc:344] Connected to a master server at 127.25.185.254:46623
I20250809 02:07:41.647377 32524 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:41.648334 32524 heartbeater.cc:507] Master 127.25.185.254:46623 requested a full tablet report, sending...
I20250809 02:07:41.650347 32069 ts_manager.cc:194] Registered new tserver with Master: fef8948af77f437388b5974a19868a02 (127.25.185.195:43325)
I20250809 02:07:41.651777 32069 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.195:56329
I20250809 02:07:41.655371 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:07:41.686708 26343 test_util.cc:276] Using random seed: 667873140
I20250809 02:07:41.726276 32068 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54184:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250809 02:07:41.728704 32068 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 02:07:41.784034 32192 tablet_service.cc:1468] Processing CreateTablet for tablet 27c7dae8669945d7a4b1ee2f823b6497 (DEFAULT_TABLE table=pre_rebuild [id=d2ff135218b344b894278ecf152a87e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:07:41.786162 32192 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27c7dae8669945d7a4b1ee2f823b6497. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:41.785601 32459 tablet_service.cc:1468] Processing CreateTablet for tablet 27c7dae8669945d7a4b1ee2f823b6497 (DEFAULT_TABLE table=pre_rebuild [id=d2ff135218b344b894278ecf152a87e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:07:41.787489 32459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27c7dae8669945d7a4b1ee2f823b6497. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:41.789427 32326 tablet_service.cc:1468] Processing CreateTablet for tablet 27c7dae8669945d7a4b1ee2f823b6497 (DEFAULT_TABLE table=pre_rebuild [id=d2ff135218b344b894278ecf152a87e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:07:41.791026 32326 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27c7dae8669945d7a4b1ee2f823b6497. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:41.810541 32548 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Bootstrap starting.
I20250809 02:07:41.813392 32549 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Bootstrap starting.
I20250809 02:07:41.815882 32550 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Bootstrap starting.
I20250809 02:07:41.820766 32549 tablet_bootstrap.cc:654] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:41.820765 32548 tablet_bootstrap.cc:654] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:41.820833 32550 tablet_bootstrap.cc:654] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:41.822474 32550 log.cc:826] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:41.823048 32548 log.cc:826] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:41.823045 32549 log.cc:826] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:41.827947 32550 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: No bootstrap required, opened a new log
I20250809 02:07:41.828452 32550 ts_tablet_manager.cc:1397] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20250809 02:07:41.828689 32549 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: No bootstrap required, opened a new log
I20250809 02:07:41.829293 32549 ts_tablet_manager.cc:1397] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.005s
I20250809 02:07:41.830415 32548 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: No bootstrap required, opened a new log
I20250809 02:07:41.830928 32548 ts_tablet_manager.cc:1397] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Time spent bootstrapping tablet: real 0.021s user 0.012s sys 0.004s
I20250809 02:07:41.845578 32550 raft_consensus.cc:357] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:41.846572 32550 raft_consensus.cc:383] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:41.846944 32550 raft_consensus.cc:738] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef8948af77f437388b5974a19868a02, State: Initialized, Role: FOLLOWER
I20250809 02:07:41.847735 32550 consensus_queue.cc:260] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:41.850914 32524 heartbeater.cc:499] Master 127.25.185.254:46623 was elected leader, sending a full tablet report...
I20250809 02:07:41.852289 32550 ts_tablet_manager.cc:1428] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Time spent starting tablet: real 0.024s user 0.017s sys 0.005s
I20250809 02:07:41.857105 32549 raft_consensus.cc:357] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:41.858282 32549 raft_consensus.cc:383] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:41.857833 32548 raft_consensus.cc:357] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:41.858624 32549 raft_consensus.cc:738] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb3097677c84677bf7892c42bf355b6, State: Initialized, Role: FOLLOWER
I20250809 02:07:41.858922 32548 raft_consensus.cc:383] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:41.859278 32548 raft_consensus.cc:738] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda3ed4d4fd8451f89c16b1ea180d2ec, State: Initialized, Role: FOLLOWER
I20250809 02:07:41.859561 32549 consensus_queue.cc:260] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:41.860700 32548 consensus_queue.cc:260] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:41.871335 32548 ts_tablet_manager.cc:1428] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Time spent starting tablet: real 0.040s user 0.022s sys 0.015s
I20250809 02:07:41.874042 32549 ts_tablet_manager.cc:1428] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Time spent starting tablet: real 0.044s user 0.031s sys 0.011s
W20250809 02:07:41.878859 32525 tablet.cc:2378] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 02:07:41.891896 32258 tablet.cc:2378] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 02:07:41.939512 32392 tablet.cc:2378] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:07:42.124763 32555 raft_consensus.cc:491] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:07:42.125247 32555 raft_consensus.cc:513] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:42.127562 32555 leader_election.cc:290] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fef8948af77f437388b5974a19868a02 (127.25.185.195:43325), fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194:42421)
I20250809 02:07:42.137921 32554 raft_consensus.cc:491] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:07:42.138538 32554 raft_consensus.cc:513] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:42.139045 32346 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27c7dae8669945d7a4b1ee2f823b6497" candidate_uuid: "4fb3097677c84677bf7892c42bf355b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" is_pre_election: true
I20250809 02:07:42.139137 32479 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27c7dae8669945d7a4b1ee2f823b6497" candidate_uuid: "4fb3097677c84677bf7892c42bf355b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef8948af77f437388b5974a19868a02" is_pre_election: true
I20250809 02:07:42.139860 32479 raft_consensus.cc:2466] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb3097677c84677bf7892c42bf355b6 in term 0.
I20250809 02:07:42.139892 32346 raft_consensus.cc:2466] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb3097677c84677bf7892c42bf355b6 in term 0.
I20250809 02:07:42.141372 32146 leader_election.cc:304] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb3097677c84677bf7892c42bf355b6, fef8948af77f437388b5974a19868a02; no voters:
I20250809 02:07:42.142230 32555 raft_consensus.cc:2802] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 02:07:42.142593 32555 raft_consensus.cc:491] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 02:07:42.142629 32554 leader_election.cc:290] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4fb3097677c84677bf7892c42bf355b6 (127.25.185.193:36219), fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194:42421)
I20250809 02:07:42.142989 32555 raft_consensus.cc:3058] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:42.149776 32555 raft_consensus.cc:513] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:42.153985 32555 leader_election.cc:290] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [CANDIDATE]: Term 1 election: Requested vote from peers fef8948af77f437388b5974a19868a02 (127.25.185.195:43325), fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194:42421)
I20250809 02:07:42.154269 32479 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27c7dae8669945d7a4b1ee2f823b6497" candidate_uuid: "4fb3097677c84677bf7892c42bf355b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef8948af77f437388b5974a19868a02"
I20250809 02:07:42.154852 32479 raft_consensus.cc:3058] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:42.155421 32346 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27c7dae8669945d7a4b1ee2f823b6497" candidate_uuid: "4fb3097677c84677bf7892c42bf355b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec"
I20250809 02:07:42.155988 32346 raft_consensus.cc:3058] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:42.156461 32345 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27c7dae8669945d7a4b1ee2f823b6497" candidate_uuid: "fef8948af77f437388b5974a19868a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" is_pre_election: true
I20250809 02:07:42.158972 32212 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27c7dae8669945d7a4b1ee2f823b6497" candidate_uuid: "fef8948af77f437388b5974a19868a02" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fb3097677c84677bf7892c42bf355b6" is_pre_election: true
I20250809 02:07:42.159996 32212 raft_consensus.cc:2391] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fef8948af77f437388b5974a19868a02 in current term 1: Already voted for candidate 4fb3097677c84677bf7892c42bf355b6 in this term.
I20250809 02:07:42.160504 32479 raft_consensus.cc:2466] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fb3097677c84677bf7892c42bf355b6 in term 1.
I20250809 02:07:42.161305 32412 leader_election.cc:304] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fef8948af77f437388b5974a19868a02; no voters: 4fb3097677c84677bf7892c42bf355b6, fda3ed4d4fd8451f89c16b1ea180d2ec
I20250809 02:07:42.161283 32146 leader_election.cc:304] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb3097677c84677bf7892c42bf355b6, fef8948af77f437388b5974a19868a02; no voters:
I20250809 02:07:42.161950 32554 raft_consensus.cc:2747] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250809 02:07:42.162350 32555 raft_consensus.cc:2802] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:07:42.162732 32346 raft_consensus.cc:2466] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fb3097677c84677bf7892c42bf355b6 in term 1.
I20250809 02:07:42.164587 32555 raft_consensus.cc:695] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 LEADER]: Becoming Leader. State: Replica: 4fb3097677c84677bf7892c42bf355b6, State: Running, Role: LEADER
I20250809 02:07:42.165409 32555 consensus_queue.cc:237] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:42.173751 32069 catalog_manager.cc:5582] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4fb3097677c84677bf7892c42bf355b6 (127.25.185.193). New cstate: current_term: 1 leader_uuid: "4fb3097677c84677bf7892c42bf355b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } health_report { overall_health: UNKNOWN } } }
I20250809 02:07:42.368316 32346 raft_consensus.cc:1273] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 1 FOLLOWER]: Refusing update from remote peer 4fb3097677c84677bf7892c42bf355b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250809 02:07:42.368500 32479 raft_consensus.cc:1273] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 1 FOLLOWER]: Refusing update from remote peer 4fb3097677c84677bf7892c42bf355b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250809 02:07:42.369870 32560 consensus_queue.cc:1035] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 02:07:42.370494 32555 consensus_queue.cc:1035] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 02:07:42.399150 32568 mvcc.cc:204] Tried to move back new op lower bound from 7187272754639781888 to 7187272753836859392. Current Snapshot: MvccSnapshot[applied={T|T < 7187272754639781888}]
I20250809 02:07:42.406713 32571 mvcc.cc:204] Tried to move back new op lower bound from 7187272754639781888 to 7187272753836859392. Current Snapshot: MvccSnapshot[applied={T|T < 7187272754639781888}]
I20250809 02:07:42.409812 32572 mvcc.cc:204] Tried to move back new op lower bound from 7187272754639781888 to 7187272753836859392. Current Snapshot: MvccSnapshot[applied={T|T < 7187272754639781888}]
W20250809 02:07:44.696492 32387 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac080 after lost signal to thread 32262
W20250809 02:07:44.697548 32387 debug-util.cc:398] Leaking SignalData structure 0x7b08000b68a0 after lost signal to thread 32390
W20250809 02:07:44.934909 32565 meta_cache.cc:1261] Time spent looking up entry by key: real 0.080s user 0.004s sys 0.000s
W20250809 02:07:47.656080 32562 meta_cache.cc:1261] Time spent looking up entry by key: real 0.067s user 0.000s sys 0.000s
I20250809 02:07:48.365072 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 32036
W20250809 02:07:48.467056 32257 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:46623 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:46623: connect: Connection refused (error 111)
W20250809 02:07:48.475023 32391 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:46623 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:46623: connect: Connection refused (error 111)
W20250809 02:07:48.495633 32524 heartbeater.cc:646] Failed to heartbeat to 127.25.185.254:46623 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.25.185.254:46623: connect: Connection refused (error 111)
W20250809 02:07:48.743423 32609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:48.744047 32609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:48.776369 32609 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250809 02:07:50.221943 32617 debug-util.cc:398] Leaking SignalData structure 0x7b08000373e0 after lost signal to thread 32609
W20250809 02:07:50.222503 32617 kernel_stack_watchdog.cc:198] Thread 32609 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 02:07:50.311650 32609 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.492s user 0.002s sys 0.000s
W20250809 02:07:50.417292 32609 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.598s user 0.005s sys 0.009s
I20250809 02:07:50.495832 32609 minidump.cc:252] Setting minidump size limit to 20M
I20250809 02:07:50.498147 32609 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:50.499243 32609 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:50.510936 32645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:50.511283 32646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:50.515213 32649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:50.516184 32609 server_base.cc:1047] running on GCE node
I20250809 02:07:50.517750 32609 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250809 02:07:50.518316 32609 hybrid_clock.cc:648] HybridClock initialized: now 1754705270518247 us; error 88255 us; skew 500 ppm
I20250809 02:07:50.519418 32609 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:50.524500 32609 webserver.cc:489] Webserver started at http://0.0.0.0:37129/ using document root <none> and password file <none>
I20250809 02:07:50.525482 32609 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:50.525717 32609 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:50.526173 32609 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250809 02:07:50.530437 32609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/instance:
uuid: "56948f378de04e58ab94dcd0ca3091c9"
format_stamp: "Formatted at 2025-08-09 02:07:50 on dist-test-slave-24ff"
I20250809 02:07:50.531635 32609 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal/instance:
uuid: "56948f378de04e58ab94dcd0ca3091c9"
format_stamp: "Formatted at 2025-08-09 02:07:50 on dist-test-slave-24ff"
I20250809 02:07:50.537568 32609 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250809 02:07:50.542428 32654 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:50.543370 32609 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250809 02:07:50.543712 32609 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
uuid: "56948f378de04e58ab94dcd0ca3091c9"
format_stamp: "Formatted at 2025-08-09 02:07:50 on dist-test-slave-24ff"
I20250809 02:07:50.544082 32609 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:50.643385 32609 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:50.644804 32609 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:50.645243 32609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:50.649937 32609 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:07:50.664191 32609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: Bootstrap starting.
I20250809 02:07:50.669318 32609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: Neither blocks nor log segments found. Creating new log.
I20250809 02:07:50.671069 32609 log.cc:826] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:50.675150 32609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: No bootstrap required, opened a new log
I20250809 02:07:50.690992 32609 raft_consensus.cc:357] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER }
I20250809 02:07:50.691579 32609 raft_consensus.cc:383] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:07:50.691807 32609 raft_consensus.cc:738] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56948f378de04e58ab94dcd0ca3091c9, State: Initialized, Role: FOLLOWER
I20250809 02:07:50.692456 32609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER }
I20250809 02:07:50.692976 32609 raft_consensus.cc:397] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:07:50.693239 32609 raft_consensus.cc:491] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:07:50.693585 32609 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:07:50.697504 32609 raft_consensus.cc:513] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER }
I20250809 02:07:50.698171 32609 leader_election.cc:304] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56948f378de04e58ab94dcd0ca3091c9; no voters:
I20250809 02:07:50.699847 32609 leader_election.cc:290] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250809 02:07:50.700124 32661 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:07:50.703287 32661 raft_consensus.cc:695] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 1 LEADER]: Becoming Leader. State: Replica: 56948f378de04e58ab94dcd0ca3091c9, State: Running, Role: LEADER
I20250809 02:07:50.704221 32661 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER }
I20250809 02:07:50.711179 32663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56948f378de04e58ab94dcd0ca3091c9. Latest consensus state: current_term: 1 leader_uuid: "56948f378de04e58ab94dcd0ca3091c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER } }
I20250809 02:07:50.711649 32662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56948f378de04e58ab94dcd0ca3091c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER } }
I20250809 02:07:50.711946 32663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [sys.catalog]: This master's current role is: LEADER
I20250809 02:07:50.712155 32662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [sys.catalog]: This master's current role is: LEADER
I20250809 02:07:50.723927 32609 tablet_replica.cc:331] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: stopping tablet replica
I20250809 02:07:50.724467 32609 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 1 LEADER]: Raft consensus shutting down.
I20250809 02:07:50.724912 32609 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250809 02:07:50.727049 32609 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250809 02:07:50.727476 32609 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250809 02:07:50.782888 32609 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250809 02:07:51.812458 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 32128
I20250809 02:07:51.857178 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 32261
I20250809 02:07:51.893464 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 32395
I20250809 02:07:51.928933 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:46623
--webserver_interface=127.25.185.254
--webserver_port=45833
--builtin_ntp_servers=127.25.185.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.25.185.254:46623 with env {}
W20250809 02:07:52.242918 32670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:52.243526 32670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:52.244027 32670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:52.276108 32670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250809 02:07:52.276458 32670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:52.276713 32670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250809 02:07:52.276963 32670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250809 02:07:52.312858 32670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.25.185.254:46623
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.25.185.254:46623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.25.185.254
--webserver_port=45833
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:52.314270 32670 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:52.315933 32670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:52.327941 32676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:52.328902 32677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:53.531924 32679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:53.534494 32678 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1201 milliseconds
I20250809 02:07:53.534610 32670 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:53.536029 32670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:53.538875 32670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:53.540223 32670 hybrid_clock.cc:648] HybridClock initialized: now 1754705273540180 us; error 60 us; skew 500 ppm
I20250809 02:07:53.541137 32670 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:53.549400 32670 webserver.cc:489] Webserver started at http://127.25.185.254:45833/ using document root <none> and password file <none>
I20250809 02:07:53.550644 32670 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:53.550921 32670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:53.559248 32670 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250809 02:07:53.564038 32686 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:53.565090 32670 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250809 02:07:53.565394 32670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
uuid: "56948f378de04e58ab94dcd0ca3091c9"
format_stamp: "Formatted at 2025-08-09 02:07:50 on dist-test-slave-24ff"
I20250809 02:07:53.567458 32670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:53.626081 32670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:53.627712 32670 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:53.628257 32670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:53.699610 32670 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.254:46623
I20250809 02:07:53.699683 32737 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.254:46623 every 8 connection(s)
I20250809 02:07:53.702356 32670 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/info.pb
I20250809 02:07:53.706015 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 32670
I20250809 02:07:53.708151 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.193:36219
--local_ip_for_outbound_sockets=127.25.185.193
--tserver_master_addrs=127.25.185.254:46623
--webserver_port=36949
--webserver_interface=127.25.185.193
--builtin_ntp_servers=127.25.185.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250809 02:07:53.717224 32738 sys_catalog.cc:263] Verifying existing consensus state
I20250809 02:07:53.730302 32738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: Bootstrap starting.
I20250809 02:07:53.740309 32738 log.cc:826] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: Log is configured to *not* fsync() on all Append() calls
I20250809 02:07:53.752367 32738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:07:53.753125 32738 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: Bootstrap complete.
I20250809 02:07:53.774371 32738 raft_consensus.cc:357] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } }
I20250809 02:07:53.775108 32738 raft_consensus.cc:738] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56948f378de04e58ab94dcd0ca3091c9, State: Initialized, Role: FOLLOWER
I20250809 02:07:53.775890 32738 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } }
I20250809 02:07:53.776383 32738 raft_consensus.cc:397] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250809 02:07:53.776660 32738 raft_consensus.cc:491] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250809 02:07:53.776976 32738 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:07:53.781078 32738 raft_consensus.cc:513] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } }
I20250809 02:07:53.781759 32738 leader_election.cc:304] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56948f378de04e58ab94dcd0ca3091c9; no voters:
I20250809 02:07:53.783946 32738 leader_election.cc:290] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250809 02:07:53.784472 32742 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 02:07:53.787662 32742 raft_consensus.cc:695] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [term 2 LEADER]: Becoming Leader. State: Replica: 56948f378de04e58ab94dcd0ca3091c9, State: Running, Role: LEADER
I20250809 02:07:53.788609 32742 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } }
I20250809 02:07:53.789328 32738 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [sys.catalog]: configured and running, proceeding with master startup.
I20250809 02:07:53.799880 32744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56948f378de04e58ab94dcd0ca3091c9. Latest consensus state: current_term: 2 leader_uuid: "56948f378de04e58ab94dcd0ca3091c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } } }
I20250809 02:07:53.800540 32744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [sys.catalog]: This master's current role is: LEADER
I20250809 02:07:53.799768 32743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "56948f378de04e58ab94dcd0ca3091c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56948f378de04e58ab94dcd0ca3091c9" member_type: VOTER last_known_addr { host: "127.25.185.254" port: 46623 } } }
I20250809 02:07:53.802177 32743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9 [sys.catalog]: This master's current role is: LEADER
I20250809 02:07:53.807785 32749 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250809 02:07:53.820849 32749 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=2ca7e7ed60e84472bf6048be0ead3f61]
I20250809 02:07:53.829121 32749 tablet_loader.cc:96] loaded metadata for tablet 27c7dae8669945d7a4b1ee2f823b6497 (table pre_rebuild [id=2ca7e7ed60e84472bf6048be0ead3f61])
I20250809 02:07:53.830989 32749 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250809 02:07:53.850065 32749 catalog_manager.cc:1349] Generated new cluster ID: fe981f6089c84141b173405d18cfc091
I20250809 02:07:53.850384 32749 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250809 02:07:53.865163 32760 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250809 02:07:53.883605 32749 catalog_manager.cc:1372] Generated new certificate authority record
I20250809 02:07:53.885761 32749 catalog_manager.cc:1506] Loading token signing keys...
I20250809 02:07:53.905184 32749 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: Generated new TSK 0
I20250809 02:07:53.906311 32749 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250809 02:07:54.058440 32740 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:54.059017 32740 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:54.059520 32740 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:54.090667 32740 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:54.091635 32740 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.193
I20250809 02:07:54.126425 32740 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.193:36219
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.25.185.193
--webserver_port=36949
--tserver_master_addrs=127.25.185.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.193
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:54.127806 32740 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:54.129429 32740 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:54.142416 32766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:55.545924 32765 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 32740
W20250809 02:07:54.143646 32767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:55.628221 300 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1482 milliseconds
W20250809 02:07:55.628336 301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:55.626032 32740 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.483s user 0.482s sys 0.999s
W20250809 02:07:55.629030 32740 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.486s user 0.483s sys 0.999s
I20250809 02:07:55.629302 32740 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:55.633534 32740 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:55.636411 32740 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:55.637948 32740 hybrid_clock.cc:648] HybridClock initialized: now 1754705275637887 us; error 50 us; skew 500 ppm
I20250809 02:07:55.639186 32740 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:55.647425 32740 webserver.cc:489] Webserver started at http://127.25.185.193:36949/ using document root <none> and password file <none>
I20250809 02:07:55.648850 32740 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:55.649205 32740 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:55.661141 32740 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.000s sys 0.009s
I20250809 02:07:55.667663 308 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:55.668965 32740 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250809 02:07:55.669421 32740 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal
uuid: "4fb3097677c84677bf7892c42bf355b6"
format_stamp: "Formatted at 2025-08-09 02:07:37 on dist-test-slave-24ff"
I20250809 02:07:55.672295 32740 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:55.734786 32740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:55.736222 32740 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:55.736651 32740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:55.739667 32740 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:55.745913 315 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 02:07:55.753640 32740 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 02:07:55.753890 32740 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.002s
I20250809 02:07:55.754156 32740 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 02:07:55.758951 32740 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 02:07:55.759212 32740 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250809 02:07:55.759593 315 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Bootstrap starting.
I20250809 02:07:55.960083 32740 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.193:36219
I20250809 02:07:55.960242 421 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.193:36219 every 8 connection(s)
I20250809 02:07:55.963826 32740 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/info.pb
I20250809 02:07:55.972980 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 32740
I20250809 02:07:55.974956 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.194:42421
--local_ip_for_outbound_sockets=127.25.185.194
--tserver_master_addrs=127.25.185.254:46623
--webserver_port=44223
--webserver_interface=127.25.185.194
--builtin_ntp_servers=127.25.185.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250809 02:07:56.005206 422 heartbeater.cc:344] Connected to a master server at 127.25.185.254:46623
I20250809 02:07:56.005784 422 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:56.007400 422 heartbeater.cc:507] Master 127.25.185.254:46623 requested a full tablet report, sending...
I20250809 02:07:56.013203 32703 ts_manager.cc:194] Registered new tserver with Master: 4fb3097677c84677bf7892c42bf355b6 (127.25.185.193:36219)
I20250809 02:07:56.021282 32703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.193:44457
I20250809 02:07:56.111649 315 log.cc:826] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Log is configured to *not* fsync() on all Append() calls
W20250809 02:07:56.309439 426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:56.309988 426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:56.310505 426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:56.342589 426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:56.343626 426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.194
I20250809 02:07:56.381055 426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.194:42421
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.25.185.194
--webserver_port=44223
--tserver_master_addrs=127.25.185.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.194
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:56.382371 426 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:56.384009 426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:56.397478 433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:57.026041 422 heartbeater.cc:499] Master 127.25.185.254:46623 was elected leader, sending a full tablet report...
W20250809 02:07:57.799471 432 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 426
W20250809 02:07:57.881125 432 kernel_stack_watchdog.cc:198] Thread 426 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 386ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250809 02:07:56.400714 434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:57.882596 426 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.472s user 0.000s sys 0.002s
W20250809 02:07:57.882974 426 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.472s user 0.001s sys 0.002s
W20250809 02:07:57.888851 435 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1487 milliseconds
W20250809 02:07:57.889454 437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:57.889556 426 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:07:57.891072 426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:07:57.893690 426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:07:57.895301 426 hybrid_clock.cc:648] HybridClock initialized: now 1754705277895265 us; error 56 us; skew 500 ppm
I20250809 02:07:57.896387 426 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:07:57.903795 426 webserver.cc:489] Webserver started at http://127.25.185.194:44223/ using document root <none> and password file <none>
I20250809 02:07:57.905031 426 fs_manager.cc:362] Metadata directory not provided
I20250809 02:07:57.905316 426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:07:57.916009 426 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20250809 02:07:57.921821 443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:07:57.923058 426 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250809 02:07:57.923467 426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal
uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec"
format_stamp: "Formatted at 2025-08-09 02:07:39 on dist-test-slave-24ff"
I20250809 02:07:57.926261 426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:07:58.011672 426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:07:58.013773 426 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:07:58.014313 426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:07:58.017627 426 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:07:58.025018 450 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 02:07:58.033039 426 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 02:07:58.033336 426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250809 02:07:58.033676 426 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 02:07:58.040922 426 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 02:07:58.041188 426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.003s sys 0.003s
I20250809 02:07:58.041508 450 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Bootstrap starting.
I20250809 02:07:58.238507 426 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.194:42421
I20250809 02:07:58.238862 556 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.194:42421 every 8 connection(s)
I20250809 02:07:58.242239 426 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/info.pb
I20250809 02:07:58.245136 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 426
I20250809 02:07:58.247105 26343 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
/tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.25.185.195:43325
--local_ip_for_outbound_sockets=127.25.185.195
--tserver_master_addrs=127.25.185.254:46623
--webserver_port=45211
--webserver_interface=127.25.185.195
--builtin_ntp_servers=127.25.185.212:43019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250809 02:07:58.295457 557 heartbeater.cc:344] Connected to a master server at 127.25.185.254:46623
I20250809 02:07:58.296170 557 heartbeater.cc:461] Registering TS with master...
I20250809 02:07:58.297719 557 heartbeater.cc:507] Master 127.25.185.254:46623 requested a full tablet report, sending...
I20250809 02:07:58.301950 32703 ts_manager.cc:194] Registered new tserver with Master: fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194:42421)
I20250809 02:07:58.304981 32703 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.194:36903
I20250809 02:07:58.379112 450 log.cc:826] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Log is configured to *not* fsync() on all Append() calls
W20250809 02:07:58.748013 561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250809 02:07:58.748677 561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250809 02:07:58.749426 561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250809 02:07:58.805260 561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250809 02:07:58.806665 561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.25.185.195
I20250809 02:07:58.868029 561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.25.185.212:43019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.25.185.195:43325
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.25.185.195
--webserver_port=45211
--tserver_master_addrs=127.25.185.254:46623
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.25.185.195
--log_dir=/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 09 Aug 2025 01:52:23 UTC on 5fd53c4cbb9d
build id 7477
TSAN enabled
I20250809 02:07:58.869731 561 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250809 02:07:58.871924 561 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250809 02:07:58.889622 569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250809 02:07:59.309263 557 heartbeater.cc:499] Master 127.25.185.254:46623 was elected leader, sending a full tablet report...
I20250809 02:07:59.449185 315 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:07:59.450830 315 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Bootstrap complete.
I20250809 02:07:59.453662 315 ts_tablet_manager.cc:1397] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Time spent bootstrapping tablet: real 3.694s user 3.510s sys 0.092s
I20250809 02:07:59.480073 315 raft_consensus.cc:357] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:59.484858 315 raft_consensus.cc:738] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb3097677c84677bf7892c42bf355b6, State: Initialized, Role: FOLLOWER
I20250809 02:07:59.486222 315 consensus_queue.cc:260] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:07:59.492592 315 ts_tablet_manager.cc:1428] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Time spent starting tablet: real 0.038s user 0.035s sys 0.005s
W20250809 02:07:58.901361 572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:07:58.899073 570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250809 02:08:00.170300 571 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250809 02:08:00.170614 561 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250809 02:08:00.174937 561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250809 02:08:00.177683 561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250809 02:08:00.179112 561 hybrid_clock.cc:648] HybridClock initialized: now 1754705280179049 us; error 79 us; skew 500 ppm
I20250809 02:08:00.179929 561 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250809 02:08:00.186946 561 webserver.cc:489] Webserver started at http://127.25.185.195:45211/ using document root <none> and password file <none>
I20250809 02:08:00.188248 561 fs_manager.cc:362] Metadata directory not provided
I20250809 02:08:00.188560 561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250809 02:08:00.199177 561 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.003s sys 0.004s
I20250809 02:08:00.205229 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250809 02:08:00.206558 561 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250809 02:08:00.207028 561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data,/tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal
uuid: "fef8948af77f437388b5974a19868a02"
format_stamp: "Formatted at 2025-08-09 02:07:41 on dist-test-slave-24ff"
I20250809 02:08:00.209802 561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250809 02:08:00.291471 561 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250809 02:08:00.293335 561 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250809 02:08:00.293797 561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250809 02:08:00.296499 561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250809 02:08:00.302577 587 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250809 02:08:00.313385 561 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250809 02:08:00.313681 561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250809 02:08:00.313975 561 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250809 02:08:00.319886 561 ts_tablet_manager.cc:610] Registered 1 tablets
I20250809 02:08:00.320158 561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.006s sys 0.000s
I20250809 02:08:00.320583 587 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Bootstrap starting.
I20250809 02:08:00.499666 561 rpc_server.cc:307] RPC server started. Bound to: 127.25.185.195:43325
I20250809 02:08:00.499809 693 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.25.185.195:43325 every 8 connection(s)
I20250809 02:08:00.503628 561 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/info.pb
I20250809 02:08:00.507854 26343 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu as pid 561
I20250809 02:08:00.585012 694 heartbeater.cc:344] Connected to a master server at 127.25.185.254:46623
I20250809 02:08:00.585544 694 heartbeater.cc:461] Registering TS with master...
I20250809 02:08:00.586998 694 heartbeater.cc:507] Master 127.25.185.254:46623 requested a full tablet report, sending...
I20250809 02:08:00.591535 32702 ts_manager.cc:194] Registered new tserver with Master: fef8948af77f437388b5974a19868a02 (127.25.185.195:43325)
I20250809 02:08:00.594429 26343 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250809 02:08:00.594897 32702 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.25.185.195:52531
I20250809 02:08:00.660629 587 log.cc:826] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Log is configured to *not* fsync() on all Append() calls
I20250809 02:08:00.747934 450 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:08:00.748731 450 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Bootstrap complete.
I20250809 02:08:00.750046 450 ts_tablet_manager.cc:1397] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Time spent bootstrapping tablet: real 2.709s user 2.631s sys 0.060s
I20250809 02:08:00.760921 450 raft_consensus.cc:357] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:08:00.763345 450 raft_consensus.cc:738] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda3ed4d4fd8451f89c16b1ea180d2ec, State: Initialized, Role: FOLLOWER
I20250809 02:08:00.764034 450 consensus_queue.cc:260] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:08:00.767127 450 ts_tablet_manager.cc:1428] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Time spent starting tablet: real 0.017s user 0.020s sys 0.000s
I20250809 02:08:00.840829 707 raft_consensus.cc:491] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:08:00.841303 707 raft_consensus.cc:513] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:08:00.843713 707 leader_election.cc:290] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fef8948af77f437388b5974a19868a02 (127.25.185.195:43325), fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194:42421)
I20250809 02:08:00.863476 512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27c7dae8669945d7a4b1ee2f823b6497" candidate_uuid: "4fb3097677c84677bf7892c42bf355b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" is_pre_election: true
I20250809 02:08:00.864425 512 raft_consensus.cc:2466] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fb3097677c84677bf7892c42bf355b6 in term 1.
I20250809 02:08:00.866117 309 leader_election.cc:304] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb3097677c84677bf7892c42bf355b6, fda3ed4d4fd8451f89c16b1ea180d2ec; no voters:
I20250809 02:08:00.867063 707 raft_consensus.cc:2802] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250809 02:08:00.867421 707 raft_consensus.cc:491] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 02:08:00.867765 707 raft_consensus.cc:3058] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Advancing to term 2
I20250809 02:08:00.862933 649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27c7dae8669945d7a4b1ee2f823b6497" candidate_uuid: "4fb3097677c84677bf7892c42bf355b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "fef8948af77f437388b5974a19868a02" is_pre_election: true
W20250809 02:08:00.874325 310 leader_election.cc:343] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fef8948af77f437388b5974a19868a02 (127.25.185.195:43325): Illegal state: must be running to vote when last-logged opid is not known
I20250809 02:08:00.876987 707 raft_consensus.cc:513] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:08:00.878803 707 leader_election.cc:290] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [CANDIDATE]: Term 2 election: Requested vote from peers fef8948af77f437388b5974a19868a02 (127.25.185.195:43325), fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194:42421)
I20250809 02:08:00.879570 649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27c7dae8669945d7a4b1ee2f823b6497" candidate_uuid: "4fb3097677c84677bf7892c42bf355b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "fef8948af77f437388b5974a19868a02"
I20250809 02:08:00.879894 512 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "27c7dae8669945d7a4b1ee2f823b6497" candidate_uuid: "4fb3097677c84677bf7892c42bf355b6" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec"
I20250809 02:08:00.880426 512 raft_consensus.cc:3058] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 1 FOLLOWER]: Advancing to term 2
W20250809 02:08:00.880534 310 leader_election.cc:343] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer fef8948af77f437388b5974a19868a02 (127.25.185.195:43325): Illegal state: must be running to vote when last-logged opid is not known
I20250809 02:08:00.889675 512 raft_consensus.cc:2466] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4fb3097677c84677bf7892c42bf355b6 in term 2.
I20250809 02:08:00.890864 309 leader_election.cc:304] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fb3097677c84677bf7892c42bf355b6, fda3ed4d4fd8451f89c16b1ea180d2ec; no voters: fef8948af77f437388b5974a19868a02
I20250809 02:08:00.891659 707 raft_consensus.cc:2802] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 2 FOLLOWER]: Leader election won for term 2
I20250809 02:08:00.893461 707 raft_consensus.cc:695] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 2 LEADER]: Becoming Leader. State: Replica: 4fb3097677c84677bf7892c42bf355b6, State: Running, Role: LEADER
I20250809 02:08:00.894419 707 consensus_queue.cc:237] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:08:00.904896 32702 catalog_manager.cc:5582] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 reported cstate change: term changed from 0 to 2, leader changed from <none> to 4fb3097677c84677bf7892c42bf355b6 (127.25.185.193), VOTER 4fb3097677c84677bf7892c42bf355b6 (127.25.185.193) added, VOTER fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194) added, VOTER fef8948af77f437388b5974a19868a02 (127.25.185.195) added. New cstate: current_term: 2 leader_uuid: "4fb3097677c84677bf7892c42bf355b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } health_report { overall_health: UNKNOWN } } }
W20250809 02:08:01.288892 26343 scanner-internal.cc:458] Time spent opening tablet: real 0.656s user 0.005s sys 0.001s
W20250809 02:08:01.295715 310 consensus_peers.cc:489] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 -> Peer fef8948af77f437388b5974a19868a02 (127.25.185.195:43325): Couldn't send request to peer fef8948af77f437388b5974a19868a02. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250809 02:08:01.431507 512 raft_consensus.cc:1273] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 2 FOLLOWER]: Refusing update from remote peer 4fb3097677c84677bf7892c42bf355b6: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250809 02:08:01.433465 707 consensus_queue.cc:1035] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250809 02:08:01.548211 377 consensus_queue.cc:237] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:08:01.554101 512 raft_consensus.cc:1273] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 2 FOLLOWER]: Refusing update from remote peer 4fb3097677c84677bf7892c42bf355b6: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250809 02:08:01.555856 723 consensus_queue.cc:1035] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250809 02:08:01.563176 723 raft_consensus.cc:2953] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER fef8948af77f437388b5974a19868a02 (127.25.185.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } }
I20250809 02:08:01.565835 512 raft_consensus.cc:2953] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER fef8948af77f437388b5974a19868a02 (127.25.185.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } }
I20250809 02:08:01.587184 32687 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 27c7dae8669945d7a4b1ee2f823b6497 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250809 02:08:01.589730 32702 catalog_manager.cc:5582] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 reported cstate change: config changed from index -1 to 207, VOTER fef8948af77f437388b5974a19868a02 (127.25.185.195) evicted. New cstate: current_term: 2 leader_uuid: "4fb3097677c84677bf7892c42bf355b6" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } health_report { overall_health: HEALTHY } } }
I20250809 02:08:01.600535 694 heartbeater.cc:499] Master 127.25.185.254:46623 was elected leader, sending a full tablet report...
I20250809 02:08:01.638098 377 consensus_queue.cc:237] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } }
I20250809 02:08:01.640837 707 raft_consensus.cc:2953] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } }
I20250809 02:08:01.649513 32687 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 27c7dae8669945d7a4b1ee2f823b6497 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250809 02:08:01.653843 32701 catalog_manager.cc:5582] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 reported cstate change: config changed from index 207 to 208, VOTER fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194) evicted. New cstate: current_term: 2 leader_uuid: "4fb3097677c84677bf7892c42bf355b6" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } health_report { overall_health: HEALTHY } } }
I20250809 02:08:01.679239 629 tablet_service.cc:1515] Processing DeleteTablet for tablet 27c7dae8669945d7a4b1ee2f823b6497 with delete_type TABLET_DATA_TOMBSTONED (TS fef8948af77f437388b5974a19868a02 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:55786
I20250809 02:08:01.693255 492 tablet_service.cc:1515] Processing DeleteTablet for tablet 27c7dae8669945d7a4b1ee2f823b6497 with delete_type TABLET_DATA_TOMBSTONED (TS fda3ed4d4fd8451f89c16b1ea180d2ec not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:47298
W20250809 02:08:01.696029 32688 catalog_manager.cc:4908] TS fef8948af77f437388b5974a19868a02 (127.25.185.195:43325): delete failed for tablet 27c7dae8669945d7a4b1ee2f823b6497 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 27c7dae8669945d7a4b1ee2f823b6497 already in progress: opening tablet
I20250809 02:08:01.705350 734 tablet_replica.cc:331] stopping tablet replica
I20250809 02:08:01.706313 734 raft_consensus.cc:2241] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 2 FOLLOWER]: Raft consensus shutting down.
I20250809 02:08:01.707185 734 raft_consensus.cc:2270] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 2 FOLLOWER]: Raft consensus is shut down!
I20250809 02:08:01.738745 734 ts_tablet_manager.cc:1905] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250809 02:08:01.754563 734 ts_tablet_manager.cc:1918] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250809 02:08:01.755230 734 log.cc:1199] T 27c7dae8669945d7a4b1ee2f823b6497 P fda3ed4d4fd8451f89c16b1ea180d2ec: Deleting WAL directory at /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/wal/wals/27c7dae8669945d7a4b1ee2f823b6497
I20250809 02:08:01.757074 32687 catalog_manager.cc:4928] TS fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194:42421): tablet 27c7dae8669945d7a4b1ee2f823b6497 (table pre_rebuild [id=2ca7e7ed60e84472bf6048be0ead3f61]) successfully deleted
I20250809 02:08:02.179670 357 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 02:08:02.203137 492 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 02:08:02.215222 629 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
56948f378de04e58ab94dcd0ca3091c9 | 127.25.185.254:46623 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.25.185.212:43019 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
4fb3097677c84677bf7892c42bf355b6 | 127.25.185.193:36219 | HEALTHY | <none> | 1 | 0
fda3ed4d4fd8451f89c16b1ea180d2ec | 127.25.185.194:42421 | HEALTHY | <none> | 0 | 0
fef8948af77f437388b5974a19868a02 | 127.25.185.195:43325 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.25.185.193 | experimental | 127.25.185.193:36219
local_ip_for_outbound_sockets | 127.25.185.194 | experimental | 127.25.185.194:42421
local_ip_for_outbound_sockets | 127.25.185.195 | experimental | 127.25.185.195:43325
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.185.193:36219
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/info.pb | hidden | 127.25.185.194:42421
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.185.195:43325
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.25.185.212:43019 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250809 02:08:02.585984 26343 log_verifier.cc:126] Checking tablet 27c7dae8669945d7a4b1ee2f823b6497
I20250809 02:08:03.096549 26343 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 27c7dae8669945d7a4b1ee2f823b6497
I20250809 02:08:03.099066 32701 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52080:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250809 02:08:03.099786 32701 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52080:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250809 02:08:03.113844 32701 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 56948f378de04e58ab94dcd0ca3091c9: Sending DeleteTablet for 1 replicas of tablet 27c7dae8669945d7a4b1ee2f823b6497
I20250809 02:08:03.116055 26343 test_util.cc:276] Using random seed: 689302527
I20250809 02:08:03.116156 357 tablet_service.cc:1515] Processing DeleteTablet for tablet 27c7dae8669945d7a4b1ee2f823b6497 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-09 02:08:03 UTC) from {username='slave'} at 127.0.0.1:54174
I20250809 02:08:03.131702 764 tablet_replica.cc:331] stopping tablet replica
I20250809 02:08:03.132629 764 raft_consensus.cc:2241] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 2 LEADER]: Raft consensus shutting down.
I20250809 02:08:03.133308 764 raft_consensus.cc:2270] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250809 02:08:03.177706 764 ts_tablet_manager.cc:1905] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250809 02:08:03.196409 764 ts_tablet_manager.cc:1918] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250809 02:08:03.197075 764 log.cc:1199] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Deleting WAL directory at /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/wal/wals/27c7dae8669945d7a4b1ee2f823b6497
I20250809 02:08:03.198345 764 ts_tablet_manager.cc:1939] T 27c7dae8669945d7a4b1ee2f823b6497 P 4fb3097677c84677bf7892c42bf355b6: Deleting consensus metadata
I20250809 02:08:03.201918 32687 catalog_manager.cc:4928] TS 4fb3097677c84677bf7892c42bf355b6 (127.25.185.193:36219): tablet 27c7dae8669945d7a4b1ee2f823b6497 (table pre_rebuild [id=2ca7e7ed60e84472bf6048be0ead3f61]) successfully deleted
I20250809 02:08:03.223037 32701 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52118:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250809 02:08:03.226641 32701 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250809 02:08:03.256858 492 tablet_service.cc:1468] Processing CreateTablet for tablet b50094dc4f9140eea695b3c2788a2db9 (DEFAULT_TABLE table=post_rebuild [id=a637da00b2b84f27b99a74149378bfdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:08:03.258266 492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b50094dc4f9140eea695b3c2788a2db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:08:03.258462 357 tablet_service.cc:1468] Processing CreateTablet for tablet b50094dc4f9140eea695b3c2788a2db9 (DEFAULT_TABLE table=post_rebuild [id=a637da00b2b84f27b99a74149378bfdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:08:03.259850 357 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b50094dc4f9140eea695b3c2788a2db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:08:03.264983 629 tablet_service.cc:1468] Processing CreateTablet for tablet b50094dc4f9140eea695b3c2788a2db9 (DEFAULT_TABLE table=post_rebuild [id=a637da00b2b84f27b99a74149378bfdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250809 02:08:03.266800 629 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b50094dc4f9140eea695b3c2788a2db9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250809 02:08:03.283281 771 tablet_bootstrap.cc:492] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6: Bootstrap starting.
I20250809 02:08:03.290848 771 tablet_bootstrap.cc:654] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6: Neither blocks nor log segments found. Creating new log.
I20250809 02:08:03.340127 771 tablet_bootstrap.cc:492] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6: No bootstrap required, opened a new log
I20250809 02:08:03.340655 771 ts_tablet_manager.cc:1397] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6: Time spent bootstrapping tablet: real 0.058s user 0.027s sys 0.026s
I20250809 02:08:03.343494 771 raft_consensus.cc:357] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } }
I20250809 02:08:03.344290 771 raft_consensus.cc:383] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:08:03.344657 771 raft_consensus.cc:738] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fb3097677c84677bf7892c42bf355b6, State: Initialized, Role: FOLLOWER
I20250809 02:08:03.345448 771 consensus_queue.cc:260] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } }
I20250809 02:08:03.351692 773 tablet_bootstrap.cc:492] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec: Bootstrap starting.
I20250809 02:08:03.355865 771 ts_tablet_manager.cc:1428] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6: Time spent starting tablet: real 0.015s user 0.006s sys 0.004s
I20250809 02:08:03.358181 773 tablet_bootstrap.cc:654] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec: Neither blocks nor log segments found. Creating new log.
I20250809 02:08:03.369881 773 tablet_bootstrap.cc:492] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec: No bootstrap required, opened a new log
I20250809 02:08:03.370317 773 ts_tablet_manager.cc:1397] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.004s
I20250809 02:08:03.372992 773 raft_consensus.cc:357] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } }
I20250809 02:08:03.373701 773 raft_consensus.cc:383] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:08:03.374032 773 raft_consensus.cc:738] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fda3ed4d4fd8451f89c16b1ea180d2ec, State: Initialized, Role: FOLLOWER
I20250809 02:08:03.374666 773 consensus_queue.cc:260] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } }
I20250809 02:08:03.380353 773 ts_tablet_manager.cc:1428] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
W20250809 02:08:03.489046 423 tablet.cc:2378] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 02:08:03.529213 558 tablet.cc:2378] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250809 02:08:03.571877 587 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250809 02:08:03.572679 587 tablet_bootstrap.cc:492] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Bootstrap complete.
I20250809 02:08:03.574018 587 ts_tablet_manager.cc:1397] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Time spent bootstrapping tablet: real 3.254s user 3.017s sys 0.088s
I20250809 02:08:03.579216 587 raft_consensus.cc:357] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:08:03.579787 587 raft_consensus.cc:738] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef8948af77f437388b5974a19868a02, State: Initialized, Role: FOLLOWER
I20250809 02:08:03.580446 587 consensus_queue.cc:260] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } }
I20250809 02:08:03.583516 587 ts_tablet_manager.cc:1428] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Time spent starting tablet: real 0.009s user 0.012s sys 0.000s
I20250809 02:08:03.584223 587 tablet_bootstrap.cc:492] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02: Bootstrap starting.
I20250809 02:08:03.589039 629 tablet_service.cc:1515] Processing DeleteTablet for tablet 27c7dae8669945d7a4b1ee2f823b6497 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-09 02:08:03 UTC) from {username='slave'} at 127.0.0.1:55786
I20250809 02:08:03.589280 587 tablet_bootstrap.cc:654] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02: Neither blocks nor log segments found. Creating new log.
I20250809 02:08:03.595432 776 raft_consensus.cc:491] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250809 02:08:03.595898 776 raft_consensus.cc:513] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } }
I20250809 02:08:03.600602 776 leader_election.cc:290] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fef8948af77f437388b5974a19868a02 (127.25.185.195:43325), 4fb3097677c84677bf7892c42bf355b6 (127.25.185.193:36219)
I20250809 02:08:03.610982 778 tablet_replica.cc:331] stopping tablet replica
I20250809 02:08:03.611977 778 raft_consensus.cc:2241] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250809 02:08:03.612646 778 raft_consensus.cc:2270] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250809 02:08:03.617733 587 tablet_bootstrap.cc:492] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02: No bootstrap required, opened a new log
I20250809 02:08:03.618494 587 ts_tablet_manager.cc:1397] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.012s
I20250809 02:08:03.626015 587 raft_consensus.cc:357] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } }
I20250809 02:08:03.626834 587 raft_consensus.cc:383] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250809 02:08:03.627235 587 raft_consensus.cc:738] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fef8948af77f437388b5974a19868a02, State: Initialized, Role: FOLLOWER
I20250809 02:08:03.627957 587 consensus_queue.cc:260] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } }
I20250809 02:08:03.632071 377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b50094dc4f9140eea695b3c2788a2db9" candidate_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fb3097677c84677bf7892c42bf355b6" is_pre_election: true
I20250809 02:08:03.632889 377 raft_consensus.cc:2466] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda3ed4d4fd8451f89c16b1ea180d2ec in term 0.
I20250809 02:08:03.634207 444 leader_election.cc:304] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fb3097677c84677bf7892c42bf355b6, fda3ed4d4fd8451f89c16b1ea180d2ec; no voters:
I20250809 02:08:03.635105 776 raft_consensus.cc:2802] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250809 02:08:03.635455 776 raft_consensus.cc:491] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250809 02:08:03.635787 776 raft_consensus.cc:3058] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:08:03.636150 587 ts_tablet_manager.cc:1428] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02: Time spent starting tablet: real 0.017s user 0.003s sys 0.004s
I20250809 02:08:03.641611 649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b50094dc4f9140eea695b3c2788a2db9" candidate_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef8948af77f437388b5974a19868a02" is_pre_election: true
I20250809 02:08:03.642339 649 raft_consensus.cc:2466] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fda3ed4d4fd8451f89c16b1ea180d2ec in term 0.
I20250809 02:08:03.647035 776 raft_consensus.cc:513] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } }
I20250809 02:08:03.649816 649 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b50094dc4f9140eea695b3c2788a2db9" candidate_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fef8948af77f437388b5974a19868a02"
I20250809 02:08:03.649963 377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b50094dc4f9140eea695b3c2788a2db9" candidate_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4fb3097677c84677bf7892c42bf355b6"
I20250809 02:08:03.650350 649 raft_consensus.cc:3058] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:08:03.650492 377 raft_consensus.cc:3058] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6 [term 0 FOLLOWER]: Advancing to term 1
I20250809 02:08:03.653692 778 ts_tablet_manager.cc:1905] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250809 02:08:03.657631 377 raft_consensus.cc:2466] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda3ed4d4fd8451f89c16b1ea180d2ec in term 1.
I20250809 02:08:03.657680 649 raft_consensus.cc:2466] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fda3ed4d4fd8451f89c16b1ea180d2ec in term 1.
I20250809 02:08:03.659113 776 leader_election.cc:290] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [CANDIDATE]: Term 1 election: Requested vote from peers fef8948af77f437388b5974a19868a02 (127.25.185.195:43325), 4fb3097677c84677bf7892c42bf355b6 (127.25.185.193:36219)
I20250809 02:08:03.658987 445 leader_election.cc:304] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fda3ed4d4fd8451f89c16b1ea180d2ec, fef8948af77f437388b5974a19868a02; no voters:
I20250809 02:08:03.662077 776 raft_consensus.cc:2802] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 1 FOLLOWER]: Leader election won for term 1
I20250809 02:08:03.662565 776 raft_consensus.cc:695] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [term 1 LEADER]: Becoming Leader. State: Replica: fda3ed4d4fd8451f89c16b1ea180d2ec, State: Running, Role: LEADER
I20250809 02:08:03.663462 776 consensus_queue.cc:237] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } } peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } }
I20250809 02:08:03.672770 32701 catalog_manager.cc:5582] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec reported cstate change: term changed from 0 to 1, leader changed from <none> to fda3ed4d4fd8451f89c16b1ea180d2ec (127.25.185.194). New cstate: current_term: 1 leader_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fda3ed4d4fd8451f89c16b1ea180d2ec" member_type: VOTER last_known_addr { host: "127.25.185.194" port: 42421 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 } health_report { overall_health: UNKNOWN } } }
I20250809 02:08:03.673602 778 ts_tablet_manager.cc:1918] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.205
I20250809 02:08:03.674041 778 log.cc:1199] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Deleting WAL directory at /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/wal/wals/27c7dae8669945d7a4b1ee2f823b6497
I20250809 02:08:03.675148 778 ts_tablet_manager.cc:1939] T 27c7dae8669945d7a4b1ee2f823b6497 P fef8948af77f437388b5974a19868a02: Deleting consensus metadata
I20250809 02:08:03.677760 32688 catalog_manager.cc:4928] TS fef8948af77f437388b5974a19868a02 (127.25.185.195:43325): tablet 27c7dae8669945d7a4b1ee2f823b6497 (table pre_rebuild [id=2ca7e7ed60e84472bf6048be0ead3f61]) successfully deleted
W20250809 02:08:03.821275 697 tablet.cc:2378] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250809 02:08:03.855203 767 meta_cache.cc:1261] Time spent looking up entry by key: real 0.055s user 0.009s sys 0.000s
W20250809 02:08:03.858656 766 meta_cache.cc:1261] Time spent looking up entry by key: real 0.058s user 0.005s sys 0.004s
I20250809 02:08:03.968139 377 raft_consensus.cc:1273] T b50094dc4f9140eea695b3c2788a2db9 P 4fb3097677c84677bf7892c42bf355b6 [term 1 FOLLOWER]: Refusing update from remote peer fda3ed4d4fd8451f89c16b1ea180d2ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250809 02:08:03.968525 649 raft_consensus.cc:1273] T b50094dc4f9140eea695b3c2788a2db9 P fef8948af77f437388b5974a19868a02 [term 1 FOLLOWER]: Refusing update from remote peer fda3ed4d4fd8451f89c16b1ea180d2ec: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250809 02:08:03.970132 776 consensus_queue.cc:1035] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fb3097677c84677bf7892c42bf355b6" member_type: VOTER last_known_addr { host: "127.25.185.193" port: 36219 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250809 02:08:03.971155 784 consensus_queue.cc:1035] T b50094dc4f9140eea695b3c2788a2db9 P fda3ed4d4fd8451f89c16b1ea180d2ec [LEADER]: Connected to new peer: Peer: permanent_uuid: "fef8948af77f437388b5974a19868a02" member_type: VOTER last_known_addr { host: "127.25.185.195" port: 43325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250809 02:08:04.008231 793 mvcc.cc:204] Tried to move back new op lower bound from 7187272843118260224 to 7187272841894973440. Current Snapshot: MvccSnapshot[applied={T|T < 7187272843118260224}]
I20250809 02:08:04.032532 796 mvcc.cc:204] Tried to move back new op lower bound from 7187272843118260224 to 7187272841894973440. Current Snapshot: MvccSnapshot[applied={T|T < 7187272843118260224}]
I20250809 02:08:08.947460 357 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250809 02:08:08.955544 492 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250809 02:08:08.960839 629 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
56948f378de04e58ab94dcd0ca3091c9 | 127.25.185.254:46623 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.25.185.212:43019 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
4fb3097677c84677bf7892c42bf355b6 | 127.25.185.193:36219 | HEALTHY | <none> | 0 | 0
fda3ed4d4fd8451f89c16b1ea180d2ec | 127.25.185.194:42421 | HEALTHY | <none> | 1 | 0
fef8948af77f437388b5974a19868a02 | 127.25.185.195:43325 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.25.185.193 | experimental | 127.25.185.193:36219
local_ip_for_outbound_sockets | 127.25.185.194 | experimental | 127.25.185.194:42421
local_ip_for_outbound_sockets | 127.25.185.195 | experimental | 127.25.185.195:43325
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-0/data/info.pb | hidden | 127.25.185.193:36219
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-1/data/info.pb | hidden | 127.25.185.194:42421
server_dump_info_path | /tmp/dist-test-taskVYoTtu/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754705097773638-26343-0/minicluster-data/ts-2/data/info.pb | hidden | 127.25.185.195:43325
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.25.185.212:43019 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250809 02:08:09.166415 26343 log_verifier.cc:126] Checking tablet 27c7dae8669945d7a4b1ee2f823b6497
I20250809 02:08:09.166908 26343 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 27c7dae8669945d7a4b1ee2f823b6497
I20250809 02:08:09.167084 26343 log_verifier.cc:126] Checking tablet b50094dc4f9140eea695b3c2788a2db9
W20250809 02:08:09.179243 418 debug-util.cc:398] Leaking SignalData structure 0x7b08000c3dc0 after lost signal to thread 32761
W20250809 02:08:09.188351 32734 debug-util.cc:398] Leaking SignalData structure 0x7b080006f340 after lost signal to thread 32671
I20250809 02:08:10.037510 26343 log_verifier.cc:177] Verified matching terms for 205 ops in tablet b50094dc4f9140eea695b3c2788a2db9
I20250809 02:08:10.068269 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 32740
I20250809 02:08:10.136391 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 426
I20250809 02:08:10.180138 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 561
I20250809 02:08:10.217851 26343 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskVYoTtu/build/tsan/bin/kudu with pid 32670
2025-08-09T02:08:10Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (36602 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (36602 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (192440 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST