Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250813 19:57:41.482910 11411 test_util.cc:276] Using random seed: -1849191138
W20250813 19:57:42.609386 11411 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.085s user 0.457s sys 0.627s
W20250813 19:57:42.609756 11411 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.086s user 0.457s sys 0.627s
I20250813 19:57:42.611734 11411 ts_itest-base.cc:115] Starting cluster with:
I20250813 19:57:42.611917 11411 ts_itest-base.cc:116] --------------
I20250813 19:57:42.612113 11411 ts_itest-base.cc:117] 4 tablet servers
I20250813 19:57:42.612272 11411 ts_itest-base.cc:118] 3 replicas per TS
I20250813 19:57:42.612412 11411 ts_itest-base.cc:119] --------------
2025-08-13T19:57:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T19:57:42Z Disabled control of system clock
I20250813 19:57:42.646661 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:43639
--webserver_interface=127.11.36.254
--webserver_port=0
--builtin_ntp_servers=127.11.36.212:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:43639 with env {}
W20250813 19:57:42.921757 11425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:57:42.922253 11425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:57:42.922610 11425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:57:42.951216 11425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 19:57:42.951511 11425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:57:42.951705 11425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 19:57:42.951900 11425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 19:57:42.983537 11425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:43639
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:57:42.984826 11425 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:57:42.986363 11425 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:57:42.997368 11431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:44.400344 11430 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 11425
W20250813 19:57:44.669811 11430 kernel_stack_watchdog.cc:198] Thread 11425 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250813 19:57:42.997485 11432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:44.673856 11425 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.677s user 0.627s sys 1.045s
W20250813 19:57:44.674193 11433 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1673 milliseconds
W20250813 19:57:44.674217 11425 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.678s user 0.627s sys 1.045s
W20250813 19:57:44.675318 11434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:57:44.675398 11425 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:57:44.676541 11425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:57:44.679280 11425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:57:44.680581 11425 hybrid_clock.cc:648] HybridClock initialized: now 1755115064680546 us; error 43 us; skew 500 ppm
I20250813 19:57:44.681336 11425 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:57:44.687403 11425 webserver.cc:480] Webserver started at http://127.11.36.254:35405/ using document root <none> and password file <none>
I20250813 19:57:44.688292 11425 fs_manager.cc:362] Metadata directory not provided
I20250813 19:57:44.688520 11425 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:57:44.688997 11425 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:57:44.693327 11425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "2427f514102f47c7bf6d9d8f22de34b5"
format_stamp: "Formatted at 2025-08-13 19:57:44 on dist-test-slave-l098"
I20250813 19:57:44.694312 11425 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "2427f514102f47c7bf6d9d8f22de34b5"
format_stamp: "Formatted at 2025-08-13 19:57:44 on dist-test-slave-l098"
I20250813 19:57:44.701022 11425 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20250813 19:57:44.706461 11441 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:44.707360 11425 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250813 19:57:44.707623 11425 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2427f514102f47c7bf6d9d8f22de34b5"
format_stamp: "Formatted at 2025-08-13 19:57:44 on dist-test-slave-l098"
I20250813 19:57:44.707928 11425 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:57:44.759788 11425 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:57:44.761217 11425 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:57:44.761631 11425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:57:44.836977 11425 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:43639
I20250813 19:57:44.837034 11492 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:43639 every 8 connection(s)
I20250813 19:57:44.839607 11425 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 19:57:44.842314 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 11425
I20250813 19:57:44.842868 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 19:57:44.845326 11493 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:57:44.865813 11493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5: Bootstrap starting.
I20250813 19:57:44.871791 11493 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5: Neither blocks nor log segments found. Creating new log.
I20250813 19:57:44.874470 11493 log.cc:826] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5: Log is configured to *not* fsync() on all Append() calls
I20250813 19:57:44.879683 11493 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5: No bootstrap required, opened a new log
I20250813 19:57:44.897569 11493 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } }
I20250813 19:57:44.898196 11493 raft_consensus.cc:383] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:57:44.898389 11493 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2427f514102f47c7bf6d9d8f22de34b5, State: Initialized, Role: FOLLOWER
I20250813 19:57:44.899118 11493 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } }
I20250813 19:57:44.899700 11493 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:57:44.899950 11493 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:57:44.900272 11493 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:57:44.904150 11493 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } }
I20250813 19:57:44.904832 11493 leader_election.cc:304] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2427f514102f47c7bf6d9d8f22de34b5; no voters:
I20250813 19:57:44.906802 11493 leader_election.cc:290] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 19:57:44.907521 11498 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:57:44.910632 11498 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 1 LEADER]: Becoming Leader. State: Replica: 2427f514102f47c7bf6d9d8f22de34b5, State: Running, Role: LEADER
I20250813 19:57:44.911279 11498 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } }
I20250813 19:57:44.911774 11493 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 19:57:44.923648 11500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2427f514102f47c7bf6d9d8f22de34b5. Latest consensus state: current_term: 1 leader_uuid: "2427f514102f47c7bf6d9d8f22de34b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } } }
I20250813 19:57:44.924293 11500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [sys.catalog]: This master's current role is: LEADER
I20250813 19:57:44.922715 11499 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2427f514102f47c7bf6d9d8f22de34b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } } }
I20250813 19:57:44.926707 11499 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [sys.catalog]: This master's current role is: LEADER
I20250813 19:57:44.932113 11508 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 19:57:44.946444 11508 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 19:57:44.961606 11508 catalog_manager.cc:1349] Generated new cluster ID: 7669852597184edc807b2fddfa4b8853
I20250813 19:57:44.961935 11508 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 19:57:44.985994 11508 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 19:57:44.987470 11508 catalog_manager.cc:1506] Loading token signing keys...
I20250813 19:57:45.007086 11508 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5: Generated new TSK 0
I20250813 19:57:45.007865 11508 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 19:57:45.025405 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:0
--local_ip_for_outbound_sockets=127.11.36.193
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:43639
--builtin_ntp_servers=127.11.36.212:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250813 19:57:45.323235 11517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:57:45.323743 11517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:57:45.324285 11517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:57:45.353621 11517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:57:45.354456 11517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 19:57:45.387233 11517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:43639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:57:45.388579 11517 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:57:45.389993 11517 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:57:45.403086 11523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:46.803815 11522 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 11517
W20250813 19:57:46.809892 11517 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.406s user 0.418s sys 0.885s
W20250813 19:57:46.810317 11517 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.407s user 0.418s sys 0.885s
W20250813 19:57:45.404104 11524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:46.813035 11526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:46.815093 11525 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1410 milliseconds
I20250813 19:57:46.815124 11517 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:57:46.816383 11517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:57:46.818372 11517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:57:46.819684 11517 hybrid_clock.cc:648] HybridClock initialized: now 1755115066819639 us; error 54 us; skew 500 ppm
I20250813 19:57:46.820400 11517 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:57:46.826519 11517 webserver.cc:480] Webserver started at http://127.11.36.193:34293/ using document root <none> and password file <none>
I20250813 19:57:46.827347 11517 fs_manager.cc:362] Metadata directory not provided
I20250813 19:57:46.827564 11517 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:57:46.827982 11517 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:57:46.832208 11517 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "3900890d60ad440f92182e0d4fc35c28"
format_stamp: "Formatted at 2025-08-13 19:57:46 on dist-test-slave-l098"
I20250813 19:57:46.833300 11517 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "3900890d60ad440f92182e0d4fc35c28"
format_stamp: "Formatted at 2025-08-13 19:57:46 on dist-test-slave-l098"
I20250813 19:57:46.840796 11517 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250813 19:57:46.846882 11533 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:46.847865 11517 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250813 19:57:46.848162 11517 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "3900890d60ad440f92182e0d4fc35c28"
format_stamp: "Formatted at 2025-08-13 19:57:46 on dist-test-slave-l098"
I20250813 19:57:46.848441 11517 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:57:46.910808 11517 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:57:46.912312 11517 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:57:46.912729 11517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:57:46.916344 11517 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:57:46.920933 11517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:57:46.921121 11517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:46.921402 11517 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:57:46.921558 11517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:47.109429 11517 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:35011
I20250813 19:57:47.109565 11645 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:35011 every 8 connection(s)
I20250813 19:57:47.112640 11517 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 19:57:47.121196 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 11517
I20250813 19:57:47.121628 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 19:57:47.126904 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.194:0
--local_ip_for_outbound_sockets=127.11.36.194
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:43639
--builtin_ntp_servers=127.11.36.212:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:57:47.140303 11646 heartbeater.cc:344] Connected to a master server at 127.11.36.254:43639
I20250813 19:57:47.140799 11646 heartbeater.cc:461] Registering TS with master...
I20250813 19:57:47.142042 11646 heartbeater.cc:507] Master 127.11.36.254:43639 requested a full tablet report, sending...
I20250813 19:57:47.145159 11458 ts_manager.cc:194] Registered new tserver with Master: 3900890d60ad440f92182e0d4fc35c28 (127.11.36.193:35011)
I20250813 19:57:47.148154 11458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:38243
W20250813 19:57:47.416390 11650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:57:47.416847 11650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:57:47.417380 11650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:57:47.448922 11650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:57:47.449658 11650 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.194
I20250813 19:57:47.479614 11650 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:43639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.194
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:57:47.480763 11650 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:57:47.482153 11650 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:57:47.494277 11656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:57:48.152271 11646 heartbeater.cc:499] Master 127.11.36.254:43639 was elected leader, sending a full tablet report...
W20250813 19:57:47.494410 11657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:48.591091 11659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:48.594556 11658 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1098 milliseconds
I20250813 19:57:48.594650 11650 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:57:48.595750 11650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:57:48.598181 11650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:57:48.599495 11650 hybrid_clock.cc:648] HybridClock initialized: now 1755115068599459 us; error 46 us; skew 500 ppm
I20250813 19:57:48.600250 11650 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:57:48.606009 11650 webserver.cc:480] Webserver started at http://127.11.36.194:46843/ using document root <none> and password file <none>
I20250813 19:57:48.606858 11650 fs_manager.cc:362] Metadata directory not provided
I20250813 19:57:48.607034 11650 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:57:48.607427 11650 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:57:48.611472 11650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c4597be1ee154989b20c895e8cdc9bfb"
format_stamp: "Formatted at 2025-08-13 19:57:48 on dist-test-slave-l098"
I20250813 19:57:48.612562 11650 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c4597be1ee154989b20c895e8cdc9bfb"
format_stamp: "Formatted at 2025-08-13 19:57:48 on dist-test-slave-l098"
I20250813 19:57:48.619009 11650 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250813 19:57:48.624570 11666 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:48.625473 11650 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250813 19:57:48.625770 11650 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c4597be1ee154989b20c895e8cdc9bfb"
format_stamp: "Formatted at 2025-08-13 19:57:48 on dist-test-slave-l098"
I20250813 19:57:48.626071 11650 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:57:48.675571 11650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:57:48.676966 11650 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:57:48.677374 11650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:57:48.680219 11650 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:57:48.683807 11650 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:57:48.684000 11650 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:48.684223 11650 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:57:48.684369 11650 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:48.826184 11650 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.194:33733
I20250813 19:57:48.826275 11778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.194:33733 every 8 connection(s)
I20250813 19:57:48.828665 11650 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 19:57:48.836670 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 11650
I20250813 19:57:48.837110 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 19:57:48.843716 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.195:0
--local_ip_for_outbound_sockets=127.11.36.195
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:43639
--builtin_ntp_servers=127.11.36.212:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:57:48.850941 11779 heartbeater.cc:344] Connected to a master server at 127.11.36.254:43639
I20250813 19:57:48.851565 11779 heartbeater.cc:461] Registering TS with master...
I20250813 19:57:48.852708 11779 heartbeater.cc:507] Master 127.11.36.254:43639 requested a full tablet report, sending...
I20250813 19:57:48.854681 11458 ts_manager.cc:194] Registered new tserver with Master: c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194:33733)
I20250813 19:57:48.855929 11458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.194:36285
W20250813 19:57:49.141027 11783 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:57:49.141495 11783 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:57:49.141957 11783 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:57:49.170461 11783 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:57:49.171190 11783 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.195
I20250813 19:57:49.202286 11783 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:43639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.195
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:57:49.203470 11783 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:57:49.204886 11783 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:57:49.216468 11789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:57:49.859083 11779 heartbeater.cc:499] Master 127.11.36.254:43639 was elected leader, sending a full tablet report...
W20250813 19:57:49.217474 11790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:50.329317 11792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:50.331508 11791 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1110 milliseconds
I20250813 19:57:50.331633 11783 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:57:50.332814 11783 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:57:50.336000 11783 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:57:50.337414 11783 hybrid_clock.cc:648] HybridClock initialized: now 1755115070337371 us; error 49 us; skew 500 ppm
I20250813 19:57:50.338142 11783 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:57:50.344627 11783 webserver.cc:480] Webserver started at http://127.11.36.195:35251/ using document root <none> and password file <none>
I20250813 19:57:50.345485 11783 fs_manager.cc:362] Metadata directory not provided
I20250813 19:57:50.345664 11783 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:57:50.346056 11783 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:57:50.350035 11783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0d4ffc712ea64ab3a9456841c006e8fd"
format_stamp: "Formatted at 2025-08-13 19:57:50 on dist-test-slave-l098"
I20250813 19:57:50.350973 11783 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0d4ffc712ea64ab3a9456841c006e8fd"
format_stamp: "Formatted at 2025-08-13 19:57:50 on dist-test-slave-l098"
I20250813 19:57:50.357240 11783 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20250813 19:57:50.362794 11799 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:50.363683 11783 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20250813 19:57:50.363984 11783 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0d4ffc712ea64ab3a9456841c006e8fd"
format_stamp: "Formatted at 2025-08-13 19:57:50 on dist-test-slave-l098"
I20250813 19:57:50.364305 11783 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:57:50.427623 11783 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:57:50.428957 11783 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:57:50.429406 11783 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:57:50.432526 11783 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:57:50.436139 11783 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:57:50.436349 11783 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:50.436535 11783 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:57:50.436712 11783 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:50.583881 11783 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.195:36751
I20250813 19:57:50.583947 11911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.195:36751 every 8 connection(s)
I20250813 19:57:50.586236 11783 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 19:57:50.594285 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 11783
I20250813 19:57:50.594751 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 19:57:50.601131 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.196:0
--local_ip_for_outbound_sockets=127.11.36.196
--webserver_interface=127.11.36.196
--webserver_port=0
--tserver_master_addrs=127.11.36.254:43639
--builtin_ntp_servers=127.11.36.212:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:57:50.612113 11912 heartbeater.cc:344] Connected to a master server at 127.11.36.254:43639
I20250813 19:57:50.612982 11912 heartbeater.cc:461] Registering TS with master...
I20250813 19:57:50.614001 11912 heartbeater.cc:507] Master 127.11.36.254:43639 requested a full tablet report, sending...
I20250813 19:57:50.616068 11458 ts_manager.cc:194] Registered new tserver with Master: 0d4ffc712ea64ab3a9456841c006e8fd (127.11.36.195:36751)
I20250813 19:57:50.617877 11458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.195:34769
W20250813 19:57:50.890051 11916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:57:50.890519 11916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:57:50.890949 11916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:57:50.921695 11916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:57:50.922617 11916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.196
I20250813 19:57:50.954655 11916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.11.36.196
--webserver_port=0
--tserver_master_addrs=127.11.36.254:43639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.196
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:57:50.955792 11916 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:57:50.957283 11916 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:57:50.969883 11922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:57:51.620929 11912 heartbeater.cc:499] Master 127.11.36.254:43639 was elected leader, sending a full tablet report...
W20250813 19:57:50.970589 11923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:52.220268 11925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:52.223332 11924 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1251 milliseconds
W20250813 19:57:52.224146 11916 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.254s user 0.421s sys 0.815s
W20250813 19:57:52.224403 11916 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.254s user 0.423s sys 0.815s
I20250813 19:57:52.224586 11916 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:57:52.225575 11916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:57:52.228368 11916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:57:52.229699 11916 hybrid_clock.cc:648] HybridClock initialized: now 1755115072229663 us; error 41 us; skew 500 ppm
I20250813 19:57:52.230396 11916 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:57:52.238637 11916 webserver.cc:480] Webserver started at http://127.11.36.196:43711/ using document root <none> and password file <none>
I20250813 19:57:52.239596 11916 fs_manager.cc:362] Metadata directory not provided
I20250813 19:57:52.239847 11916 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:57:52.240391 11916 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:57:52.244457 11916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "cf9f046ebd264a9cb2874ec850bb15d5"
format_stamp: "Formatted at 2025-08-13 19:57:52 on dist-test-slave-l098"
I20250813 19:57:52.245461 11916 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "cf9f046ebd264a9cb2874ec850bb15d5"
format_stamp: "Formatted at 2025-08-13 19:57:52 on dist-test-slave-l098"
I20250813 19:57:52.253098 11916 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250813 19:57:52.259441 11933 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:52.260462 11916 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20250813 19:57:52.260756 11916 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "cf9f046ebd264a9cb2874ec850bb15d5"
format_stamp: "Formatted at 2025-08-13 19:57:52 on dist-test-slave-l098"
I20250813 19:57:52.261044 11916 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:57:52.321517 11916 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:57:52.322867 11916 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:57:52.323263 11916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:57:52.326040 11916 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:57:52.329761 11916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:57:52.329932 11916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:52.330183 11916 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:57:52.330319 11916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:52.477144 11916 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.196:33055
I20250813 19:57:52.477254 12045 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.196:33055 every 8 connection(s)
I20250813 19:57:52.479491 11916 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250813 19:57:52.482978 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 11916
I20250813 19:57:52.483502 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250813 19:57:52.504401 12046 heartbeater.cc:344] Connected to a master server at 127.11.36.254:43639
I20250813 19:57:52.504917 12046 heartbeater.cc:461] Registering TS with master...
I20250813 19:57:52.506036 12046 heartbeater.cc:507] Master 127.11.36.254:43639 requested a full tablet report, sending...
I20250813 19:57:52.507998 11458 ts_manager.cc:194] Registered new tserver with Master: cf9f046ebd264a9cb2874ec850bb15d5 (127.11.36.196:33055)
I20250813 19:57:52.509601 11458 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.196:46801
I20250813 19:57:52.520105 11411 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250813 19:57:52.557796 11458 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49950:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250813 19:57:52.631171 11981 tablet_service.cc:1468] Processing CreateTablet for tablet d53ca4a5832649668677e5e165b1697a (DEFAULT_TABLE table=TestTable [id=8d587203a4184f69848e49d0d53a991c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:57:52.633924 11981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d53ca4a5832649668677e5e165b1697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:57:52.635463 11847 tablet_service.cc:1468] Processing CreateTablet for tablet d53ca4a5832649668677e5e165b1697a (DEFAULT_TABLE table=TestTable [id=8d587203a4184f69848e49d0d53a991c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:57:52.637087 11847 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d53ca4a5832649668677e5e165b1697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:57:52.637559 11714 tablet_service.cc:1468] Processing CreateTablet for tablet d53ca4a5832649668677e5e165b1697a (DEFAULT_TABLE table=TestTable [id=8d587203a4184f69848e49d0d53a991c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:57:52.639199 11714 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d53ca4a5832649668677e5e165b1697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:57:52.660599 12065 tablet_bootstrap.cc:492] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb: Bootstrap starting.
I20250813 19:57:52.663602 12067 tablet_bootstrap.cc:492] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd: Bootstrap starting.
I20250813 19:57:52.665904 12066 tablet_bootstrap.cc:492] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5: Bootstrap starting.
I20250813 19:57:52.669415 12065 tablet_bootstrap.cc:654] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb: Neither blocks nor log segments found. Creating new log.
I20250813 19:57:52.671507 12067 tablet_bootstrap.cc:654] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd: Neither blocks nor log segments found. Creating new log.
I20250813 19:57:52.671619 12066 tablet_bootstrap.cc:654] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5: Neither blocks nor log segments found. Creating new log.
I20250813 19:57:52.671762 12065 log.cc:826] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb: Log is configured to *not* fsync() on all Append() calls
I20250813 19:57:52.673784 12066 log.cc:826] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5: Log is configured to *not* fsync() on all Append() calls
I20250813 19:57:52.674101 12067 log.cc:826] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd: Log is configured to *not* fsync() on all Append() calls
I20250813 19:57:52.679291 12065 tablet_bootstrap.cc:492] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb: No bootstrap required, opened a new log
I20250813 19:57:52.680266 12067 tablet_bootstrap.cc:492] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd: No bootstrap required, opened a new log
I20250813 19:57:52.680326 12065 ts_tablet_manager.cc:1397] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.012s
I20250813 19:57:52.680507 12066 tablet_bootstrap.cc:492] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5: No bootstrap required, opened a new log
I20250813 19:57:52.680634 12067 ts_tablet_manager.cc:1397] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.008s
I20250813 19:57:52.680938 12066 ts_tablet_manager.cc:1397] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.008s
I20250813 19:57:52.698751 12067 raft_consensus.cc:357] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:52.699391 12067 raft_consensus.cc:383] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:57:52.699630 12067 raft_consensus.cc:738] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d4ffc712ea64ab3a9456841c006e8fd, State: Initialized, Role: FOLLOWER
I20250813 19:57:52.700338 12067 consensus_queue.cc:260] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:52.704613 12067 ts_tablet_manager.cc:1428] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd: Time spent starting tablet: real 0.024s user 0.020s sys 0.004s
I20250813 19:57:52.706769 12066 raft_consensus.cc:357] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:52.706945 12065 raft_consensus.cc:357] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:52.707685 12066 raft_consensus.cc:383] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:57:52.707710 12065 raft_consensus.cc:383] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:57:52.707965 12065 raft_consensus.cc:738] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4597be1ee154989b20c895e8cdc9bfb, State: Initialized, Role: FOLLOWER
I20250813 19:57:52.707967 12066 raft_consensus.cc:738] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf9f046ebd264a9cb2874ec850bb15d5, State: Initialized, Role: FOLLOWER
I20250813 19:57:52.708761 12065 consensus_queue.cc:260] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:52.708827 12066 consensus_queue.cc:260] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:52.715183 12046 heartbeater.cc:499] Master 127.11.36.254:43639 was elected leader, sending a full tablet report...
I20250813 19:57:52.717368 12066 ts_tablet_manager.cc:1428] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5: Time spent starting tablet: real 0.036s user 0.034s sys 0.002s
I20250813 19:57:52.719909 12065 ts_tablet_manager.cc:1428] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb: Time spent starting tablet: real 0.039s user 0.030s sys 0.009s
W20250813 19:57:52.740046 12047 tablet.cc:2378] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 19:57:52.837776 11780 tablet.cc:2378] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 19:57:52.842895 11913 tablet.cc:2378] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:57:52.941402 12073 raft_consensus.cc:491] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 19:57:52.941865 12073 raft_consensus.cc:513] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:52.944234 12073 leader_election.cc:290] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf9f046ebd264a9cb2874ec850bb15d5 (127.11.36.196:33055), 0d4ffc712ea64ab3a9456841c006e8fd (127.11.36.195:36751)
I20250813 19:57:52.954412 12001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d53ca4a5832649668677e5e165b1697a" candidate_uuid: "c4597be1ee154989b20c895e8cdc9bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" is_pre_election: true
I20250813 19:57:52.954411 11867 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d53ca4a5832649668677e5e165b1697a" candidate_uuid: "c4597be1ee154989b20c895e8cdc9bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" is_pre_election: true
I20250813 19:57:52.955083 11867 raft_consensus.cc:2466] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4597be1ee154989b20c895e8cdc9bfb in term 0.
I20250813 19:57:52.955083 12001 raft_consensus.cc:2466] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c4597be1ee154989b20c895e8cdc9bfb in term 0.
I20250813 19:57:52.956133 11668 leader_election.cc:304] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d4ffc712ea64ab3a9456841c006e8fd, c4597be1ee154989b20c895e8cdc9bfb; no voters:
I20250813 19:57:52.956779 12073 raft_consensus.cc:2802] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 19:57:52.957019 12073 raft_consensus.cc:491] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 19:57:52.957273 12073 raft_consensus.cc:3058] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:57:52.961301 12073 raft_consensus.cc:513] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:52.962600 12073 leader_election.cc:290] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [CANDIDATE]: Term 1 election: Requested vote from peers cf9f046ebd264a9cb2874ec850bb15d5 (127.11.36.196:33055), 0d4ffc712ea64ab3a9456841c006e8fd (127.11.36.195:36751)
I20250813 19:57:52.963172 12001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d53ca4a5832649668677e5e165b1697a" candidate_uuid: "c4597be1ee154989b20c895e8cdc9bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf9f046ebd264a9cb2874ec850bb15d5"
I20250813 19:57:52.963557 12001 raft_consensus.cc:3058] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:57:52.963686 11867 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d53ca4a5832649668677e5e165b1697a" candidate_uuid: "c4597be1ee154989b20c895e8cdc9bfb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d4ffc712ea64ab3a9456841c006e8fd"
I20250813 19:57:52.964184 11867 raft_consensus.cc:3058] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:57:52.969934 12001 raft_consensus.cc:2466] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4597be1ee154989b20c895e8cdc9bfb in term 1.
I20250813 19:57:52.970350 11867 raft_consensus.cc:2466] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c4597be1ee154989b20c895e8cdc9bfb in term 1.
I20250813 19:57:52.970949 11669 leader_election.cc:304] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c4597be1ee154989b20c895e8cdc9bfb, cf9f046ebd264a9cb2874ec850bb15d5; no voters:
I20250813 19:57:52.971499 12073 raft_consensus.cc:2802] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:57:52.972986 12073 raft_consensus.cc:695] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [term 1 LEADER]: Becoming Leader. State: Replica: c4597be1ee154989b20c895e8cdc9bfb, State: Running, Role: LEADER
I20250813 19:57:52.973743 12073 consensus_queue.cc:237] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:52.984009 11457 catalog_manager.cc:5582] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb reported cstate change: term changed from 0 to 1, leader changed from <none> to c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194). New cstate: current_term: 1 leader_uuid: "c4597be1ee154989b20c895e8cdc9bfb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } health_report { overall_health: UNKNOWN } } }
I20250813 19:57:53.084262 11411 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250813 19:57:53.087612 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c4597be1ee154989b20c895e8cdc9bfb to finish bootstrapping
I20250813 19:57:53.099373 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0d4ffc712ea64ab3a9456841c006e8fd to finish bootstrapping
I20250813 19:57:53.109696 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cf9f046ebd264a9cb2874ec850bb15d5 to finish bootstrapping
I20250813 19:57:53.119578 11411 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250813 19:57:53.122392 11411 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "d53ca4a5832649668677e5e165b1697a"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb"
rpc_addresses {
host: "127.11.36.194"
port: 33733
}
}
ts_infos {
permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5"
rpc_addresses {
host: "127.11.36.196"
port: 33055
}
}
ts_infos {
permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd"
rpc_addresses {
host: "127.11.36.195"
port: 36751
}
}
I20250813 19:57:53.421548 12073 consensus_queue.cc:1035] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 19:57:53.438052 12077 consensus_queue.cc:1035] T d53ca4a5832649668677e5e165b1697a P c4597be1ee154989b20c895e8cdc9bfb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 19:57:53.446039 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 11650
W20250813 19:57:53.473737 11801 connection.cc:537] server connection from 127.11.36.194:45857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250813 19:57:53.475191 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 11425
I20250813 19:57:53.502646 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:43639
--webserver_interface=127.11.36.254
--webserver_port=35405
--builtin_ntp_servers=127.11.36.212:42099
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:43639 with env {}
W20250813 19:57:53.783939 12089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:57:53.784490 12089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:57:53.784907 12089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:57:53.813992 12089 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 19:57:53.814316 12089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:57:53.814561 12089 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 19:57:53.814818 12089 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 19:57:53.849817 12089 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:42099
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:43639
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:43639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=35405
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:57:53.851097 12089 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:57:53.852767 12089 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:57:53.862507 12095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:54.182332 11646 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:43639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:43639: connect: Connection refused (error 111)
W20250813 19:57:54.456154 12046 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:43639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:43639: connect: Connection refused (error 111)
W20250813 19:57:54.469305 11912 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:43639 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:43639: connect: Connection refused (error 111)
I20250813 19:57:54.936722 12104 raft_consensus.cc:491] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c4597be1ee154989b20c895e8cdc9bfb)
I20250813 19:57:54.937209 12104 raft_consensus.cc:513] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
W20250813 19:57:54.942538 11936 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.36.194:33733: connect: Connection refused (error 111)
I20250813 19:57:54.946276 12107 raft_consensus.cc:491] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c4597be1ee154989b20c895e8cdc9bfb)
I20250813 19:57:54.946795 12104 leader_election.cc:290] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194:33733), 0d4ffc712ea64ab3a9456841c006e8fd (127.11.36.195:36751)
I20250813 19:57:54.947907 12107 raft_consensus.cc:513] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:54.951854 12107 leader_election.cc:290] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194:33733), cf9f046ebd264a9cb2874ec850bb15d5 (127.11.36.196:33055)
W20250813 19:57:54.955988 11802 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.36.194:33733: connect: Connection refused (error 111)
I20250813 19:57:54.978117 11867 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d53ca4a5832649668677e5e165b1697a" candidate_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" is_pre_election: true
I20250813 19:57:54.978606 11867 raft_consensus.cc:2466] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf9f046ebd264a9cb2874ec850bb15d5 in term 1.
W20250813 19:57:54.979468 11802 leader_election.cc:336] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194:33733): Network error: Client connection negotiation failed: client connection to 127.11.36.194:33733: connect: Connection refused (error 111)
I20250813 19:57:54.980616 11935 leader_election.cc:304] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d4ffc712ea64ab3a9456841c006e8fd, cf9f046ebd264a9cb2874ec850bb15d5; no voters:
I20250813 19:57:54.981434 12104 raft_consensus.cc:2802] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250813 19:57:54.981738 12104 raft_consensus.cc:491] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader c4597be1ee154989b20c895e8cdc9bfb)
W20250813 19:57:54.981897 11936 leader_election.cc:336] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194:33733): Network error: Client connection negotiation failed: client connection to 127.11.36.194:33733: connect: Connection refused (error 111)
I20250813 19:57:54.982085 12104 raft_consensus.cc:3058] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 1 FOLLOWER]: Advancing to term 2
I20250813 19:57:54.990701 12104 raft_consensus.cc:513] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:54.994056 12001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d53ca4a5832649668677e5e165b1697a" candidate_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" is_pre_election: true
I20250813 19:57:54.994071 11867 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d53ca4a5832649668677e5e165b1697a" candidate_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "0d4ffc712ea64ab3a9456841c006e8fd"
I20250813 19:57:54.994626 11867 raft_consensus.cc:3058] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 1 FOLLOWER]: Advancing to term 2
I20250813 19:57:54.994786 12001 raft_consensus.cc:2391] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 0d4ffc712ea64ab3a9456841c006e8fd in current term 2: Already voted for candidate cf9f046ebd264a9cb2874ec850bb15d5 in this term.
I20250813 19:57:54.996101 11802 leader_election.cc:304] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d4ffc712ea64ab3a9456841c006e8fd; no voters: c4597be1ee154989b20c895e8cdc9bfb, cf9f046ebd264a9cb2874ec850bb15d5
W20250813 19:57:53.864084 12096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:57:54.998915 12097 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1134 milliseconds
W20250813 19:57:54.999006 12098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:57:54.999516 12089 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:57:54.999990 12104 leader_election.cc:290] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [CANDIDATE]: Term 2 election: Requested vote from peers c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194:33733), 0d4ffc712ea64ab3a9456841c006e8fd (127.11.36.195:36751)
I20250813 19:57:55.001186 11867 raft_consensus.cc:2466] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf9f046ebd264a9cb2874ec850bb15d5 in term 2.
I20250813 19:57:55.001577 12107 raft_consensus.cc:2747] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250813 19:57:55.002522 11936 leader_election.cc:336] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194:33733): Network error: Client connection negotiation failed: client connection to 127.11.36.194:33733: connect: Connection refused (error 111)
I20250813 19:57:55.004040 12089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:57:55.003993 11935 leader_election.cc:304] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0d4ffc712ea64ab3a9456841c006e8fd, cf9f046ebd264a9cb2874ec850bb15d5; no voters: c4597be1ee154989b20c895e8cdc9bfb
I20250813 19:57:55.004642 12104 raft_consensus.cc:2802] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 19:57:55.006458 12104 raft_consensus.cc:695] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 2 LEADER]: Becoming Leader. State: Replica: cf9f046ebd264a9cb2874ec850bb15d5, State: Running, Role: LEADER
I20250813 19:57:55.007236 12089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:57:55.007395 12104 consensus_queue.cc:237] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:55.008692 12089 hybrid_clock.cc:648] HybridClock initialized: now 1755115075008660 us; error 43 us; skew 500 ppm
I20250813 19:57:55.009430 12089 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:57:55.015359 12089 webserver.cc:480] Webserver started at http://127.11.36.254:35405/ using document root <none> and password file <none>
I20250813 19:57:55.016240 12089 fs_manager.cc:362] Metadata directory not provided
I20250813 19:57:55.016474 12089 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:57:55.023003 12089 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20250813 19:57:55.026894 12118 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:57:55.027778 12089 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250813 19:57:55.028059 12089 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "2427f514102f47c7bf6d9d8f22de34b5"
format_stamp: "Formatted at 2025-08-13 19:57:44 on dist-test-slave-l098"
I20250813 19:57:55.029698 12089 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:57:55.073810 12089 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:57:55.075037 12089 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:57:55.075421 12089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:57:55.140936 12089 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:43639
I20250813 19:57:55.140992 12169 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:43639 every 8 connection(s)
I20250813 19:57:55.143424 12089 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 19:57:55.148191 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 12089
I20250813 19:57:55.148622 11411 kudu-admin-test.cc:735] Forcing unsafe config change on tserver cf9f046ebd264a9cb2874ec850bb15d5
I20250813 19:57:55.152333 12170 sys_catalog.cc:263] Verifying existing consensus state
I20250813 19:57:55.157461 12170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5: Bootstrap starting.
I20250813 19:57:55.204267 12170 log.cc:826] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5: Log is configured to *not* fsync() on all Append() calls
I20250813 19:57:55.212729 11646 heartbeater.cc:344] Connected to a master server at 127.11.36.254:43639
I20250813 19:57:55.230818 12170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:57:55.231745 12170 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5: Bootstrap complete.
I20250813 19:57:55.257527 12170 raft_consensus.cc:357] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } }
I20250813 19:57:55.260586 12170 raft_consensus.cc:738] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2427f514102f47c7bf6d9d8f22de34b5, State: Initialized, Role: FOLLOWER
I20250813 19:57:55.261502 12170 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } }
I20250813 19:57:55.262248 12170 raft_consensus.cc:397] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:57:55.262580 12170 raft_consensus.cc:491] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:57:55.262954 12170 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 1 FOLLOWER]: Advancing to term 2
I20250813 19:57:55.271953 12170 raft_consensus.cc:513] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } }
I20250813 19:57:55.272743 12170 leader_election.cc:304] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2427f514102f47c7bf6d9d8f22de34b5; no voters:
I20250813 19:57:55.274516 12170 leader_election.cc:290] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250813 19:57:55.274711 12176 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 19:57:55.277653 12176 raft_consensus.cc:695] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [term 2 LEADER]: Becoming Leader. State: Replica: 2427f514102f47c7bf6d9d8f22de34b5, State: Running, Role: LEADER
I20250813 19:57:55.278339 12176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } }
I20250813 19:57:55.278960 12170 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 19:57:55.287315 12177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2427f514102f47c7bf6d9d8f22de34b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } } }
I20250813 19:57:55.287993 12177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [sys.catalog]: This master's current role is: LEADER
I20250813 19:57:55.290221 12178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2427f514102f47c7bf6d9d8f22de34b5. Latest consensus state: current_term: 2 leader_uuid: "2427f514102f47c7bf6d9d8f22de34b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2427f514102f47c7bf6d9d8f22de34b5" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 43639 } } }
I20250813 19:57:55.290925 12178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5 [sys.catalog]: This master's current role is: LEADER
I20250813 19:57:55.301699 12184 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 19:57:55.313735 12184 catalog_manager.cc:671] Loaded metadata for table TestTable [id=8d587203a4184f69848e49d0d53a991c]
I20250813 19:57:55.320938 12184 tablet_loader.cc:96] loaded metadata for tablet d53ca4a5832649668677e5e165b1697a (table TestTable [id=8d587203a4184f69848e49d0d53a991c])
I20250813 19:57:55.322532 12184 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 19:57:55.327231 12184 catalog_manager.cc:1261] Loaded cluster ID: 7669852597184edc807b2fddfa4b8853
I20250813 19:57:55.327548 12184 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 19:57:55.334067 12184 catalog_manager.cc:1506] Loading token signing keys...
I20250813 19:57:55.338253 12184 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 2427f514102f47c7bf6d9d8f22de34b5: Loaded TSK: 0
I20250813 19:57:55.339635 12184 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 19:57:55.481755 11912 heartbeater.cc:344] Connected to a master server at 127.11.36.254:43639
I20250813 19:57:55.487385 12135 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" instance_seqno: 1755115070551198) as {username='slave'} at 127.11.36.195:40219; Asking this server to re-register.
I20250813 19:57:55.488835 11912 heartbeater.cc:461] Registering TS with master...
I20250813 19:57:55.489329 11912 heartbeater.cc:507] Master 127.11.36.254:43639 requested a full tablet report, sending...
I20250813 19:57:55.492161 12134 ts_manager.cc:194] Registered new tserver with Master: 0d4ffc712ea64ab3a9456841c006e8fd (127.11.36.195:36751)
I20250813 19:57:55.495363 11867 raft_consensus.cc:1273] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 2 FOLLOWER]: Refusing update from remote peer cf9f046ebd264a9cb2874ec850bb15d5: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250813 19:57:55.496872 12104 consensus_queue.cc:1035] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250813 19:57:55.514330 12134 catalog_manager.cc:5582] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd reported cstate change: term changed from 1 to 2, leader changed from c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194) to cf9f046ebd264a9cb2874ec850bb15d5 (127.11.36.196). New cstate: current_term: 2 leader_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } }
W20250813 19:57:55.516232 12172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:57:55.517000 12172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:57:55.562608 11936 consensus_peers.cc:489] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 -> Peer c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194:33733): Couldn't send request to peer c4597be1ee154989b20c895e8cdc9bfb. Status: Network error: Client connection negotiation failed: client connection to 127.11.36.194:33733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250813 19:57:55.569283 12046 heartbeater.cc:344] Connected to a master server at 127.11.36.254:43639
I20250813 19:57:55.572988 12135 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" instance_seqno: 1755115072443708) as {username='slave'} at 127.11.36.196:58873; Asking this server to re-register.
W20250813 19:57:55.574739 12172 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250813 19:57:55.574864 12046 heartbeater.cc:461] Registering TS with master...
I20250813 19:57:55.575416 12046 heartbeater.cc:507] Master 127.11.36.254:43639 requested a full tablet report, sending...
I20250813 19:57:55.578188 12134 ts_manager.cc:194] Registered new tserver with Master: cf9f046ebd264a9cb2874ec850bb15d5 (127.11.36.196:33055)
I20250813 19:57:56.218518 12134 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" instance_seqno: 1755115067064995) as {username='slave'} at 127.11.36.193:33973; Asking this server to re-register.
I20250813 19:57:56.221036 11646 heartbeater.cc:461] Registering TS with master...
I20250813 19:57:56.222117 11646 heartbeater.cc:507] Master 127.11.36.254:43639 requested a full tablet report, sending...
I20250813 19:57:56.225114 12134 ts_manager.cc:194] Registered new tserver with Master: 3900890d60ad440f92182e0d4fc35c28 (127.11.36.193:35011)
W20250813 19:57:57.017887 12172 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.396s user 0.449s sys 0.861s
W20250813 19:57:57.018213 12172 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.397s user 0.450s sys 0.862s
I20250813 19:57:57.084666 12001 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "cf9f046ebd264a9cb2874ec850bb15d5"
tablet_id: "d53ca4a5832649668677e5e165b1697a"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd"
}
peers {
permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5"
}
}
from {username='slave'} at 127.0.0.1:41034
W20250813 19:57:57.086119 12001 raft_consensus.cc:2216] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } unsafe_config_change: true
I20250813 19:57:57.087306 12001 raft_consensus.cc:3053] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 2 LEADER]: Stepping down as leader of term 2
I20250813 19:57:57.087632 12001 raft_consensus.cc:738] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: cf9f046ebd264a9cb2874ec850bb15d5, State: Running, Role: LEADER
I20250813 19:57:57.088397 12001 consensus_queue.cc:260] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:57.089661 12001 raft_consensus.cc:3058] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 2 FOLLOWER]: Advancing to term 3
I20250813 19:57:58.119781 12229 raft_consensus.cc:491] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 2 FOLLOWER]: Starting pre-election (detected failure of leader cf9f046ebd264a9cb2874ec850bb15d5)
I20250813 19:57:58.120154 12229 raft_consensus.cc:513] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } }
I20250813 19:57:58.121299 12229 leader_election.cc:290] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194:33733), cf9f046ebd264a9cb2874ec850bb15d5 (127.11.36.196:33055)
I20250813 19:57:58.123153 12001 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d53ca4a5832649668677e5e165b1697a" candidate_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" is_pre_election: true
W20250813 19:57:58.125284 11802 leader_election.cc:336] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194:33733): Network error: Client connection negotiation failed: client connection to 127.11.36.194:33733: connect: Connection refused (error 111)
I20250813 19:57:58.125556 11802 leader_election.cc:304] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0d4ffc712ea64ab3a9456841c006e8fd; no voters: c4597be1ee154989b20c895e8cdc9bfb, cf9f046ebd264a9cb2874ec850bb15d5
I20250813 19:57:58.126037 12229 raft_consensus.cc:2747] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250813 19:57:58.597924 12234 raft_consensus.cc:491] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250813 19:57:58.598320 12234 raft_consensus.cc:513] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } unsafe_config_change: true
I20250813 19:57:58.599167 12234 leader_election.cc:290] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 0d4ffc712ea64ab3a9456841c006e8fd (127.11.36.195:36751)
I20250813 19:57:58.599969 11867 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d53ca4a5832649668677e5e165b1697a" candidate_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" is_pre_election: true
I20250813 19:57:58.600414 11867 raft_consensus.cc:2466] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf9f046ebd264a9cb2874ec850bb15d5 in term 2.
I20250813 19:57:58.601112 11935 leader_election.cc:304] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d4ffc712ea64ab3a9456841c006e8fd, cf9f046ebd264a9cb2874ec850bb15d5; no voters:
I20250813 19:57:58.601615 12234 raft_consensus.cc:2802] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250813 19:57:58.601851 12234 raft_consensus.cc:491] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250813 19:57:58.602080 12234 raft_consensus.cc:3058] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 3 FOLLOWER]: Advancing to term 4
I20250813 19:57:58.605774 12234 raft_consensus.cc:513] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } unsafe_config_change: true
I20250813 19:57:58.606740 12234 leader_election.cc:290] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [CANDIDATE]: Term 4 election: Requested vote from peers 0d4ffc712ea64ab3a9456841c006e8fd (127.11.36.195:36751)
I20250813 19:57:58.607601 11867 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d53ca4a5832649668677e5e165b1697a" candidate_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "0d4ffc712ea64ab3a9456841c006e8fd"
I20250813 19:57:58.608016 11867 raft_consensus.cc:3058] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 2 FOLLOWER]: Advancing to term 4
I20250813 19:57:58.611896 11867 raft_consensus.cc:2466] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf9f046ebd264a9cb2874ec850bb15d5 in term 4.
I20250813 19:57:58.612780 11935 leader_election.cc:304] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 0d4ffc712ea64ab3a9456841c006e8fd, cf9f046ebd264a9cb2874ec850bb15d5; no voters:
I20250813 19:57:58.613312 12234 raft_consensus.cc:2802] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 4 FOLLOWER]: Leader election won for term 4
I20250813 19:57:58.614009 12234 raft_consensus.cc:695] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 4 LEADER]: Becoming Leader. State: Replica: cf9f046ebd264a9cb2874ec850bb15d5, State: Running, Role: LEADER
I20250813 19:57:58.614571 12234 consensus_queue.cc:237] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } unsafe_config_change: true
I20250813 19:57:58.619323 12135 catalog_manager.cc:5582] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER cf9f046ebd264a9cb2874ec850bb15d5 (127.11.36.196), VOTER 0d4ffc712ea64ab3a9456841c006e8fd (127.11.36.195). New cstate: current_term: 4 leader_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c4597be1ee154989b20c895e8cdc9bfb" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 33733 } } peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } unsafe_config_change: true }
I20250813 19:57:59.054669 11867 raft_consensus.cc:1273] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 4 FOLLOWER]: Refusing update from remote peer cf9f046ebd264a9cb2874ec850bb15d5: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250813 19:57:59.055686 12234 consensus_queue.cc:1035] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250813 19:57:59.062206 12235 raft_consensus.cc:2953] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } unsafe_config_change: true }
I20250813 19:57:59.063589 11867 raft_consensus.cc:2953] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } unsafe_config_change: true }
I20250813 19:57:59.074970 12135 catalog_manager.cc:5582] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 reported cstate change: config changed from index -1 to 3, VOTER c4597be1ee154989b20c895e8cdc9bfb (127.11.36.194) evicted, no longer has a pending config: VOTER cf9f046ebd264a9cb2874ec850bb15d5 (127.11.36.196), VOTER 0d4ffc712ea64ab3a9456841c006e8fd (127.11.36.195). New cstate: current_term: 4 leader_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250813 19:57:59.081568 12135 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet d53ca4a5832649668677e5e165b1697a on TS c4597be1ee154989b20c895e8cdc9bfb: Not found: failed to reset TS proxy: Could not find TS for UUID c4597be1ee154989b20c895e8cdc9bfb
I20250813 19:57:59.095623 12001 consensus_queue.cc:237] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: true } } unsafe_config_change: true
I20250813 19:57:59.101027 11867 raft_consensus.cc:1273] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 4 FOLLOWER]: Refusing update from remote peer cf9f046ebd264a9cb2874ec850bb15d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250813 19:57:59.101938 12234 consensus_queue.cc:1035] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250813 19:57:59.107067 12235 raft_consensus.cc:2953] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 3900890d60ad440f92182e0d4fc35c28 (127.11.36.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: true } } unsafe_config_change: true }
I20250813 19:57:59.109606 11867 raft_consensus.cc:2953] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 3900890d60ad440f92182e0d4fc35c28 (127.11.36.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: true } } unsafe_config_change: true }
W20250813 19:57:59.109902 11935 consensus_peers.cc:489] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 -> Peer 3900890d60ad440f92182e0d4fc35c28 (127.11.36.193:35011): Couldn't send request to peer 3900890d60ad440f92182e0d4fc35c28. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d53ca4a5832649668677e5e165b1697a. This is attempt 1: this message will repeat every 5th retry.
I20250813 19:57:59.115515 12121 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet d53ca4a5832649668677e5e165b1697a with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250813 19:57:59.118635 12134 catalog_manager.cc:5582] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd reported cstate change: config changed from index 3 to 5, NON_VOTER 3900890d60ad440f92182e0d4fc35c28 (127.11.36.193) added. New cstate: current_term: 4 leader_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: true } } unsafe_config_change: true }
W20250813 19:57:59.134119 12120 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet d53ca4a5832649668677e5e165b1697a on TS c4597be1ee154989b20c895e8cdc9bfb failed: Not found: failed to reset TS proxy: Could not find TS for UUID c4597be1ee154989b20c895e8cdc9bfb
I20250813 19:57:59.545594 12250 ts_tablet_manager.cc:927] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: Initiating tablet copy from peer cf9f046ebd264a9cb2874ec850bb15d5 (127.11.36.196:33055)
I20250813 19:57:59.547895 12250 tablet_copy_client.cc:323] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: tablet copy: Beginning tablet copy session from remote peer at address 127.11.36.196:33055
I20250813 19:57:59.556988 12021 tablet_copy_service.cc:140] P cf9f046ebd264a9cb2874ec850bb15d5: Received BeginTabletCopySession request for tablet d53ca4a5832649668677e5e165b1697a from peer 3900890d60ad440f92182e0d4fc35c28 ({username='slave'} at 127.11.36.193:48653)
I20250813 19:57:59.557412 12021 tablet_copy_service.cc:161] P cf9f046ebd264a9cb2874ec850bb15d5: Beginning new tablet copy session on tablet d53ca4a5832649668677e5e165b1697a from peer 3900890d60ad440f92182e0d4fc35c28 at {username='slave'} at 127.11.36.193:48653: session id = 3900890d60ad440f92182e0d4fc35c28-d53ca4a5832649668677e5e165b1697a
I20250813 19:57:59.562101 12021 tablet_copy_source_session.cc:215] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5: Tablet Copy: opened 0 blocks and 1 log segments
I20250813 19:57:59.566627 12250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d53ca4a5832649668677e5e165b1697a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:57:59.583838 12250 tablet_copy_client.cc:806] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: tablet copy: Starting download of 0 data blocks...
I20250813 19:57:59.584368 12250 tablet_copy_client.cc:670] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: tablet copy: Starting download of 1 WAL segments...
I20250813 19:57:59.588100 12250 tablet_copy_client.cc:538] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250813 19:57:59.593155 12250 tablet_bootstrap.cc:492] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: Bootstrap starting.
I20250813 19:57:59.604360 12250 log.cc:826] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: Log is configured to *not* fsync() on all Append() calls
I20250813 19:57:59.614519 12250 tablet_bootstrap.cc:492] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:57:59.615173 12250 tablet_bootstrap.cc:492] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: Bootstrap complete.
I20250813 19:57:59.615726 12250 ts_tablet_manager.cc:1397] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: Time spent bootstrapping tablet: real 0.023s user 0.019s sys 0.000s
I20250813 19:57:59.631369 12250 raft_consensus.cc:357] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: true } } unsafe_config_change: true
I20250813 19:57:59.632160 12250 raft_consensus.cc:738] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 3900890d60ad440f92182e0d4fc35c28, State: Initialized, Role: LEARNER
I20250813 19:57:59.632700 12250 consensus_queue.cc:260] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: true } } unsafe_config_change: true
I20250813 19:57:59.635668 12250 ts_tablet_manager.cc:1428] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28: Time spent starting tablet: real 0.020s user 0.019s sys 0.004s
I20250813 19:57:59.637336 12021 tablet_copy_service.cc:342] P cf9f046ebd264a9cb2874ec850bb15d5: Request end of tablet copy session 3900890d60ad440f92182e0d4fc35c28-d53ca4a5832649668677e5e165b1697a received from {username='slave'} at 127.11.36.193:48653
I20250813 19:57:59.637737 12021 tablet_copy_service.cc:434] P cf9f046ebd264a9cb2874ec850bb15d5: ending tablet copy session 3900890d60ad440f92182e0d4fc35c28-d53ca4a5832649668677e5e165b1697a on tablet d53ca4a5832649668677e5e165b1697a with peer 3900890d60ad440f92182e0d4fc35c28
I20250813 19:58:00.070693 11601 raft_consensus.cc:1215] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250813 19:58:00.300199 12120 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet d53ca4a5832649668677e5e165b1697a on TS c4597be1ee154989b20c895e8cdc9bfb failed: Not found: failed to reset TS proxy: Could not find TS for UUID c4597be1ee154989b20c895e8cdc9bfb
I20250813 19:58:00.679347 12257 raft_consensus.cc:1062] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5: attempting to promote NON_VOTER 3900890d60ad440f92182e0d4fc35c28 to VOTER
I20250813 19:58:00.680634 12257 consensus_queue.cc:237] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: false } } unsafe_config_change: true
I20250813 19:58:00.684381 11601 raft_consensus.cc:1273] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28 [term 4 LEARNER]: Refusing update from remote peer cf9f046ebd264a9cb2874ec850bb15d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250813 19:58:00.685030 11867 raft_consensus.cc:1273] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 4 FOLLOWER]: Refusing update from remote peer cf9f046ebd264a9cb2874ec850bb15d5: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250813 19:58:00.685433 12257 consensus_queue.cc:1035] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250813 19:58:00.686079 12258 consensus_queue.cc:1035] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250813 19:58:00.692178 12257 raft_consensus.cc:2953] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 3900890d60ad440f92182e0d4fc35c28 (127.11.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: false } } unsafe_config_change: true }
I20250813 19:58:00.693480 11601 raft_consensus.cc:2953] T d53ca4a5832649668677e5e165b1697a P 3900890d60ad440f92182e0d4fc35c28 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 3900890d60ad440f92182e0d4fc35c28 (127.11.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: false } } unsafe_config_change: true }
I20250813 19:58:00.693442 11867 raft_consensus.cc:2953] T d53ca4a5832649668677e5e165b1697a P 0d4ffc712ea64ab3a9456841c006e8fd [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 3900890d60ad440f92182e0d4fc35c28 (127.11.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: false } } unsafe_config_change: true }
I20250813 19:58:00.704535 12134 catalog_manager.cc:5582] T d53ca4a5832649668677e5e165b1697a P cf9f046ebd264a9cb2874ec850bb15d5 reported cstate change: config changed from index 5 to 6, 3900890d60ad440f92182e0d4fc35c28 (127.11.36.193) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5" member_type: VOTER last_known_addr { host: "127.11.36.196" port: 33055 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3900890d60ad440f92182e0d4fc35c28" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 35011 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250813 19:58:00.797752 11411 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250813 19:58:00.808135 11411 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "d53ca4a5832649668677e5e165b1697a"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "cf9f046ebd264a9cb2874ec850bb15d5"
rpc_addresses {
host: "127.11.36.196"
port: 33055
}
}
ts_infos {
permanent_uuid: "0d4ffc712ea64ab3a9456841c006e8fd"
rpc_addresses {
host: "127.11.36.195"
port: 36751
}
}
ts_infos {
permanent_uuid: "3900890d60ad440f92182e0d4fc35c28"
rpc_addresses {
host: "127.11.36.193"
port: 35011
}
}
I20250813 19:58:00.810110 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 11517
I20250813 19:58:00.832099 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 11783
I20250813 19:58:00.854682 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 11916
I20250813 19:58:00.880837 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 12089
2025-08-13T19:58:00Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19457 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250813 19:58:00.938258 11411 test_util.cc:276] Using random seed: -1829735700
I20250813 19:58:00.943514 11411 ts_itest-base.cc:115] Starting cluster with:
I20250813 19:58:00.943677 11411 ts_itest-base.cc:116] --------------
I20250813 19:58:00.943810 11411 ts_itest-base.cc:117] 3 tablet servers
I20250813 19:58:00.943931 11411 ts_itest-base.cc:118] 3 replicas per TS
I20250813 19:58:00.944042 11411 ts_itest-base.cc:119] --------------
2025-08-13T19:58:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T19:58:00Z Disabled control of system clock
I20250813 19:58:00.978704 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:45831
--webserver_interface=127.11.36.254
--webserver_port=0
--builtin_ntp_servers=127.11.36.212:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:45831
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250813 19:58:01.248637 12278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:01.249130 12278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:01.249495 12278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:01.278388 12278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 19:58:01.278635 12278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:01.278816 12278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 19:58:01.278990 12278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 19:58:01.312528 12278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:45831
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:45831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:01.313681 12278 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:01.315078 12278 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:01.326009 12284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:02.729385 12283 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 12278
W20250813 19:58:01.326023 12285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:02.987918 12278 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.662s user 0.619s sys 1.042s
W20250813 19:58:02.988797 12278 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.663s user 0.620s sys 1.042s
W20250813 19:58:02.990254 12287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:02.992233 12286 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1662 milliseconds
I20250813 19:58:02.992254 12278 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:02.993348 12278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:02.995537 12278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:02.996973 12278 hybrid_clock.cc:648] HybridClock initialized: now 1755115082996932 us; error 40 us; skew 500 ppm
I20250813 19:58:02.997764 12278 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:03.003218 12278 webserver.cc:480] Webserver started at http://127.11.36.254:40175/ using document root <none> and password file <none>
I20250813 19:58:03.004027 12278 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:03.004271 12278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:03.004671 12278 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:03.008860 12278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3103c56c41cb4f94ab307218504f6816"
format_stamp: "Formatted at 2025-08-13 19:58:03 on dist-test-slave-l098"
I20250813 19:58:03.009835 12278 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3103c56c41cb4f94ab307218504f6816"
format_stamp: "Formatted at 2025-08-13 19:58:03 on dist-test-slave-l098"
I20250813 19:58:03.016160 12278 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20250813 19:58:03.021157 12295 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:03.022056 12278 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.005s
I20250813 19:58:03.022331 12278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3103c56c41cb4f94ab307218504f6816"
format_stamp: "Formatted at 2025-08-13 19:58:03 on dist-test-slave-l098"
I20250813 19:58:03.022588 12278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:03.074062 12278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:03.075413 12278 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:03.075860 12278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:03.139015 12278 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:45831
I20250813 19:58:03.139082 12346 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:45831 every 8 connection(s)
I20250813 19:58:03.141654 12278 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 19:58:03.145556 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 12278
I20250813 19:58:03.146088 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 19:58:03.146270 12347 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:03.170394 12347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816: Bootstrap starting.
I20250813 19:58:03.175464 12347 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:03.176995 12347 log.cc:826] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:03.181113 12347 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816: No bootstrap required, opened a new log
I20250813 19:58:03.197182 12347 raft_consensus.cc:357] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3103c56c41cb4f94ab307218504f6816" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 45831 } }
I20250813 19:58:03.197821 12347 raft_consensus.cc:383] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:03.198031 12347 raft_consensus.cc:738] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3103c56c41cb4f94ab307218504f6816, State: Initialized, Role: FOLLOWER
I20250813 19:58:03.198778 12347 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3103c56c41cb4f94ab307218504f6816" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 45831 } }
I20250813 19:58:03.199304 12347 raft_consensus.cc:397] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:58:03.199512 12347 raft_consensus.cc:491] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:58:03.199740 12347 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:03.203248 12347 raft_consensus.cc:513] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3103c56c41cb4f94ab307218504f6816" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 45831 } }
I20250813 19:58:03.203862 12347 leader_election.cc:304] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3103c56c41cb4f94ab307218504f6816; no voters:
I20250813 19:58:03.205332 12347 leader_election.cc:290] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 19:58:03.206053 12352 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:03.208027 12352 raft_consensus.cc:695] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [term 1 LEADER]: Becoming Leader. State: Replica: 3103c56c41cb4f94ab307218504f6816, State: Running, Role: LEADER
I20250813 19:58:03.208835 12352 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3103c56c41cb4f94ab307218504f6816" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 45831 } }
I20250813 19:58:03.209185 12347 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 19:58:03.217427 12354 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3103c56c41cb4f94ab307218504f6816. Latest consensus state: current_term: 1 leader_uuid: "3103c56c41cb4f94ab307218504f6816" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3103c56c41cb4f94ab307218504f6816" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 45831 } } }
I20250813 19:58:03.218371 12354 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [sys.catalog]: This master's current role is: LEADER
I20250813 19:58:03.220844 12353 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3103c56c41cb4f94ab307218504f6816" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3103c56c41cb4f94ab307218504f6816" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 45831 } } }
I20250813 19:58:03.222352 12353 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816 [sys.catalog]: This master's current role is: LEADER
I20250813 19:58:03.223968 12361 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 19:58:03.232991 12361 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 19:58:03.248412 12361 catalog_manager.cc:1349] Generated new cluster ID: df14380f241c412e9ea3ef4bf35d1bfe
I20250813 19:58:03.248713 12361 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 19:58:03.279415 12361 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 19:58:03.280978 12361 catalog_manager.cc:1506] Loading token signing keys...
I20250813 19:58:03.293134 12361 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 3103c56c41cb4f94ab307218504f6816: Generated new TSK 0
I20250813 19:58:03.294088 12361 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 19:58:03.314100 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:0
--local_ip_for_outbound_sockets=127.11.36.193
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:45831
--builtin_ntp_servers=127.11.36.212:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250813 19:58:03.592520 12371 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250813 19:58:03.593087 12371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:03.593314 12371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:03.593735 12371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:03.622717 12371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:03.623544 12371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 19:58:03.657123 12371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:45831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:03.658355 12371 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:03.659857 12371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:03.672086 12377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:03.673214 12378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:05.395519 12371 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.722s user 0.656s sys 1.024s
W20250813 19:58:05.396006 12371 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.723s user 0.657s sys 1.024s
W20250813 19:58:05.074985 12376 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 12371
W20250813 19:58:05.400537 12379 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1727 milliseconds
W20250813 19:58:05.401240 12380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:58:05.401294 12371 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:05.402488 12371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:05.404434 12371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:05.405738 12371 hybrid_clock.cc:648] HybridClock initialized: now 1755115085405711 us; error 29 us; skew 500 ppm
I20250813 19:58:05.406471 12371 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:05.412693 12371 webserver.cc:480] Webserver started at http://127.11.36.193:46875/ using document root <none> and password file <none>
I20250813 19:58:05.413527 12371 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:05.413707 12371 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:05.414081 12371 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:05.418097 12371 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "dc47bb2a10f74261afb8ed36e751a73d"
format_stamp: "Formatted at 2025-08-13 19:58:05 on dist-test-slave-l098"
I20250813 19:58:05.419147 12371 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "dc47bb2a10f74261afb8ed36e751a73d"
format_stamp: "Formatted at 2025-08-13 19:58:05 on dist-test-slave-l098"
I20250813 19:58:05.425525 12371 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250813 19:58:05.431380 12387 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:05.432576 12371 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250813 19:58:05.432917 12371 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "dc47bb2a10f74261afb8ed36e751a73d"
format_stamp: "Formatted at 2025-08-13 19:58:05 on dist-test-slave-l098"
I20250813 19:58:05.433254 12371 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:05.492918 12371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:05.494338 12371 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:05.494755 12371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:05.497545 12371 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:05.501730 12371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:05.501914 12371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:05.502132 12371 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:05.502272 12371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:05.657732 12371 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:40577
I20250813 19:58:05.657884 12499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:40577 every 8 connection(s)
I20250813 19:58:05.660049 12371 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 19:58:05.663522 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 12371
I20250813 19:58:05.663978 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 19:58:05.673360 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.194:0
--local_ip_for_outbound_sockets=127.11.36.194
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:45831
--builtin_ntp_servers=127.11.36.212:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250813 19:58:05.701953 12500 heartbeater.cc:344] Connected to a master server at 127.11.36.254:45831
I20250813 19:58:05.702458 12500 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:05.703648 12500 heartbeater.cc:507] Master 127.11.36.254:45831 requested a full tablet report, sending...
I20250813 19:58:05.706503 12312 ts_manager.cc:194] Registered new tserver with Master: dc47bb2a10f74261afb8ed36e751a73d (127.11.36.193:40577)
I20250813 19:58:05.709265 12312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:36021
W20250813 19:58:05.965556 12504 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250813 19:58:05.966233 12504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:05.966507 12504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:05.966992 12504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:05.996922 12504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:05.997681 12504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.194
I20250813 19:58:06.030575 12504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:45831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.194
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:06.031703 12504 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:06.033094 12504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:06.044775 12510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:58:06.713335 12500 heartbeater.cc:499] Master 127.11.36.254:45831 was elected leader, sending a full tablet report...
W20250813 19:58:06.048306 12513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:06.045748 12511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:07.158615 12512 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1109 milliseconds
I20250813 19:58:07.158751 12504 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:07.159902 12504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:07.161965 12504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:07.163295 12504 hybrid_clock.cc:648] HybridClock initialized: now 1755115087163242 us; error 64 us; skew 500 ppm
I20250813 19:58:07.164039 12504 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:07.170117 12504 webserver.cc:480] Webserver started at http://127.11.36.194:34439/ using document root <none> and password file <none>
I20250813 19:58:07.170957 12504 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:07.171145 12504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:07.171553 12504 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:07.175539 12504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ce62b68d7392410fb8d42e3f3f11d552"
format_stamp: "Formatted at 2025-08-13 19:58:07 on dist-test-slave-l098"
I20250813 19:58:07.176537 12504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ce62b68d7392410fb8d42e3f3f11d552"
format_stamp: "Formatted at 2025-08-13 19:58:07 on dist-test-slave-l098"
I20250813 19:58:07.183264 12504 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250813 19:58:07.188450 12520 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:07.189368 12504 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250813 19:58:07.189635 12504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ce62b68d7392410fb8d42e3f3f11d552"
format_stamp: "Formatted at 2025-08-13 19:58:07 on dist-test-slave-l098"
I20250813 19:58:07.189906 12504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:07.244829 12504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:07.246296 12504 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:07.246738 12504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:07.249434 12504 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:07.253290 12504 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:07.253484 12504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:07.253710 12504 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:07.253929 12504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250813 19:58:07.376681 12504 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.194:34549
I20250813 19:58:07.376768 12632 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.194:34549 every 8 connection(s)
I20250813 19:58:07.379004 12504 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 19:58:07.385712 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 12504
I20250813 19:58:07.386195 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 19:58:07.391726 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.195:0
--local_ip_for_outbound_sockets=127.11.36.195
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:45831
--builtin_ntp_servers=127.11.36.212:35477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250813 19:58:07.402786 12633 heartbeater.cc:344] Connected to a master server at 127.11.36.254:45831
I20250813 19:58:07.403168 12633 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:07.404084 12633 heartbeater.cc:507] Master 127.11.36.254:45831 requested a full tablet report, sending...
I20250813 19:58:07.405947 12312 ts_manager.cc:194] Registered new tserver with Master: ce62b68d7392410fb8d42e3f3f11d552 (127.11.36.194:34549)
I20250813 19:58:07.407152 12312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.194:36153
W20250813 19:58:07.678057 12637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250813 19:58:07.678684 12637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:07.678990 12637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:07.679462 12637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:07.707796 12637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:07.708633 12637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.195
I20250813 19:58:07.740711 12637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:45831
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.195
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:07.741950 12637 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:07.743402 12637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:07.754106 12643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:58:08.410151 12633 heartbeater.cc:499] Master 127.11.36.254:45831 was elected leader, sending a full tablet report...
W20250813 19:58:07.754860 12644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:07.758651 12646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:58:07.758942 12637 server_base.cc:1047] running on GCE node
I20250813 19:58:08.840373 12637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:08.842778 12637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:08.844187 12637 hybrid_clock.cc:648] HybridClock initialized: now 1755115088844132 us; error 80 us; skew 500 ppm
I20250813 19:58:08.844900 12637 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:08.851198 12637 webserver.cc:480] Webserver started at http://127.11.36.195:39123/ using document root <none> and password file <none>
I20250813 19:58:08.851956 12637 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:08.852164 12637 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:08.852572 12637 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:08.856869 12637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3594f0d25e9f41a0ac9170f179db3dce"
format_stamp: "Formatted at 2025-08-13 19:58:08 on dist-test-slave-l098"
I20250813 19:58:08.857821 12637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3594f0d25e9f41a0ac9170f179db3dce"
format_stamp: "Formatted at 2025-08-13 19:58:08 on dist-test-slave-l098"
I20250813 19:58:08.864012 12637 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250813 19:58:08.869330 12653 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:08.870189 12637 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250813 19:58:08.870450 12637 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3594f0d25e9f41a0ac9170f179db3dce"
format_stamp: "Formatted at 2025-08-13 19:58:08 on dist-test-slave-l098"
I20250813 19:58:08.870728 12637 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:08.933025 12637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:08.934551 12637 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:08.935004 12637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:08.937242 12637 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:08.941072 12637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:08.941275 12637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:08.941497 12637 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:08.941666 12637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:09.062338 12637 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.195:41095
I20250813 19:58:09.062438 12765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.195:41095 every 8 connection(s)
I20250813 19:58:09.064694 12637 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 19:58:09.066416 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 12637
I20250813 19:58:09.066926 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 19:58:09.084141 12766 heartbeater.cc:344] Connected to a master server at 127.11.36.254:45831
I20250813 19:58:09.084564 12766 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:09.085497 12766 heartbeater.cc:507] Master 127.11.36.254:45831 requested a full tablet report, sending...
I20250813 19:58:09.087725 12312 ts_manager.cc:194] Registered new tserver with Master: 3594f0d25e9f41a0ac9170f179db3dce (127.11.36.195:41095)
I20250813 19:58:09.088819 12312 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.195:60769
I20250813 19:58:09.100618 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 19:58:09.132210 12312 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55766:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250813 19:58:09.149621 12312 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 19:58:09.197461 12701 tablet_service.cc:1468] Processing CreateTablet for tablet 5fd83c2c55a04c8dadd09f4fdbc6c40f (DEFAULT_TABLE table=TestTable [id=83f6a67ef51349f29007a681af8c29b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:09.199256 12701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fd83c2c55a04c8dadd09f4fdbc6c40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:09.201231 12568 tablet_service.cc:1468] Processing CreateTablet for tablet 5fd83c2c55a04c8dadd09f4fdbc6c40f (DEFAULT_TABLE table=TestTable [id=83f6a67ef51349f29007a681af8c29b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:09.202852 12568 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fd83c2c55a04c8dadd09f4fdbc6c40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:09.207983 12435 tablet_service.cc:1468] Processing CreateTablet for tablet 5fd83c2c55a04c8dadd09f4fdbc6c40f (DEFAULT_TABLE table=TestTable [id=83f6a67ef51349f29007a681af8c29b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:09.209623 12435 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fd83c2c55a04c8dadd09f4fdbc6c40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:09.225850 12785 tablet_bootstrap.cc:492] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce: Bootstrap starting.
I20250813 19:58:09.227947 12786 tablet_bootstrap.cc:492] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552: Bootstrap starting.
I20250813 19:58:09.233264 12785 tablet_bootstrap.cc:654] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:09.233217 12787 tablet_bootstrap.cc:492] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d: Bootstrap starting.
I20250813 19:58:09.233547 12786 tablet_bootstrap.cc:654] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:09.234997 12785 log.cc:826] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:09.235626 12786 log.cc:826] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:09.239930 12785 tablet_bootstrap.cc:492] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce: No bootstrap required, opened a new log
I20250813 19:58:09.240228 12787 tablet_bootstrap.cc:654] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:09.240322 12785 ts_tablet_manager.cc:1397] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20250813 19:58:09.241765 12786 tablet_bootstrap.cc:492] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552: No bootstrap required, opened a new log
I20250813 19:58:09.242177 12786 ts_tablet_manager.cc:1397] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250813 19:58:09.242156 12787 log.cc:826] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:09.249362 12787 tablet_bootstrap.cc:492] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d: No bootstrap required, opened a new log
I20250813 19:58:09.249823 12787 ts_tablet_manager.cc:1397] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20250813 19:58:09.257323 12785 raft_consensus.cc:357] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
I20250813 19:58:09.257912 12785 raft_consensus.cc:738] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3594f0d25e9f41a0ac9170f179db3dce, State: Initialized, Role: FOLLOWER
I20250813 19:58:09.258507 12785 consensus_queue.cc:260] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
I20250813 19:58:09.264413 12766 heartbeater.cc:499] Master 127.11.36.254:45831 was elected leader, sending a full tablet report...
I20250813 19:58:09.265300 12785 ts_tablet_manager.cc:1428] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce: Time spent starting tablet: real 0.025s user 0.009s sys 0.014s
I20250813 19:58:09.266683 12786 raft_consensus.cc:357] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
I20250813 19:58:09.267618 12786 raft_consensus.cc:738] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce62b68d7392410fb8d42e3f3f11d552, State: Initialized, Role: FOLLOWER
I20250813 19:58:09.268388 12786 consensus_queue.cc:260] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
I20250813 19:58:09.272325 12786 ts_tablet_manager.cc:1428] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552: Time spent starting tablet: real 0.030s user 0.028s sys 0.004s
I20250813 19:58:09.273880 12787 raft_consensus.cc:357] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
I20250813 19:58:09.274688 12787 raft_consensus.cc:738] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc47bb2a10f74261afb8ed36e751a73d, State: Initialized, Role: FOLLOWER
I20250813 19:58:09.275287 12787 consensus_queue.cc:260] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
I20250813 19:58:09.278851 12787 ts_tablet_manager.cc:1428] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d: Time spent starting tablet: real 0.029s user 0.029s sys 0.000s
I20250813 19:58:09.291864 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 19:58:09.294757 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver dc47bb2a10f74261afb8ed36e751a73d to finish bootstrapping
I20250813 19:58:09.306547 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ce62b68d7392410fb8d42e3f3f11d552 to finish bootstrapping
I20250813 19:58:09.315697 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3594f0d25e9f41a0ac9170f179db3dce to finish bootstrapping
W20250813 19:58:09.318912 12767 tablet.cc:2378] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:58:09.352380 12455 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5fd83c2c55a04c8dadd09f4fdbc6c40f"
dest_uuid: "dc47bb2a10f74261afb8ed36e751a73d"
from {username='slave'} at 127.0.0.1:59622
I20250813 19:58:09.352901 12455 raft_consensus.cc:491] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250813 19:58:09.353168 12455 raft_consensus.cc:3058] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:09.357290 12455 raft_consensus.cc:513] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
I20250813 19:58:09.359335 12455 leader_election.cc:290] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [CANDIDATE]: Term 1 election: Requested vote from peers 3594f0d25e9f41a0ac9170f179db3dce (127.11.36.195:41095), ce62b68d7392410fb8d42e3f3f11d552 (127.11.36.194:34549)
I20250813 19:58:09.367408 11411 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250813 19:58:09.370913 12721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fd83c2c55a04c8dadd09f4fdbc6c40f" candidate_uuid: "dc47bb2a10f74261afb8ed36e751a73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3594f0d25e9f41a0ac9170f179db3dce"
I20250813 19:58:09.371459 12721 raft_consensus.cc:3058] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:09.371451 12588 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fd83c2c55a04c8dadd09f4fdbc6c40f" candidate_uuid: "dc47bb2a10f74261afb8ed36e751a73d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ce62b68d7392410fb8d42e3f3f11d552"
I20250813 19:58:09.371955 12588 raft_consensus.cc:3058] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:09.375880 12588 raft_consensus.cc:2466] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc47bb2a10f74261afb8ed36e751a73d in term 1.
I20250813 19:58:09.376207 12721 raft_consensus.cc:2466] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc47bb2a10f74261afb8ed36e751a73d in term 1.
I20250813 19:58:09.376870 12388 leader_election.cc:304] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ce62b68d7392410fb8d42e3f3f11d552, dc47bb2a10f74261afb8ed36e751a73d; no voters:
I20250813 19:58:09.377732 12793 raft_consensus.cc:2802] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:09.379745 12793 raft_consensus.cc:695] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 1 LEADER]: Becoming Leader. State: Replica: dc47bb2a10f74261afb8ed36e751a73d, State: Running, Role: LEADER
I20250813 19:58:09.380563 12793 consensus_queue.cc:237] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
W20250813 19:58:09.384819 12634 tablet.cc:2378] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:58:09.391824 12309 catalog_manager.cc:5582] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d reported cstate change: term changed from 0 to 1, leader changed from <none> to dc47bb2a10f74261afb8ed36e751a73d (127.11.36.193). New cstate: current_term: 1 leader_uuid: "dc47bb2a10f74261afb8ed36e751a73d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } health_report { overall_health: HEALTHY } } }
W20250813 19:58:09.435561 12501 tablet.cc:2378] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:58:09.472848 11411 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250813 19:58:09.678189 11411 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250813 19:58:09.783684 12793 consensus_queue.cc:1035] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 19:58:09.797010 12793 consensus_queue.cc:1035] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 19:58:11.517333 12455 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5fd83c2c55a04c8dadd09f4fdbc6c40f"
dest_uuid: "dc47bb2a10f74261afb8ed36e751a73d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59632
I20250813 19:58:11.517743 12455 raft_consensus.cc:604] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 1 LEADER]: Received request to transfer leadership
I20250813 19:58:11.716113 12831 raft_consensus.cc:991] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d: : Instructing follower ce62b68d7392410fb8d42e3f3f11d552 to start an election
I20250813 19:58:11.716462 12818 raft_consensus.cc:1079] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 1 LEADER]: Signalling peer ce62b68d7392410fb8d42e3f3f11d552 to start an election
I20250813 19:58:11.717662 12588 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "5fd83c2c55a04c8dadd09f4fdbc6c40f"
dest_uuid: "ce62b68d7392410fb8d42e3f3f11d552"
from {username='slave'} at 127.11.36.193:33121
I20250813 19:58:11.718120 12588 raft_consensus.cc:491] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250813 19:58:11.718348 12588 raft_consensus.cc:3058] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [term 1 FOLLOWER]: Advancing to term 2
I20250813 19:58:11.722071 12588 raft_consensus.cc:513] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
I20250813 19:58:11.723902 12588 leader_election.cc:290] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [CANDIDATE]: Term 2 election: Requested vote from peers 3594f0d25e9f41a0ac9170f179db3dce (127.11.36.195:41095), dc47bb2a10f74261afb8ed36e751a73d (127.11.36.193:40577)
I20250813 19:58:11.732853 12721 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fd83c2c55a04c8dadd09f4fdbc6c40f" candidate_uuid: "ce62b68d7392410fb8d42e3f3f11d552" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "3594f0d25e9f41a0ac9170f179db3dce"
I20250813 19:58:11.733243 12721 raft_consensus.cc:3058] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce [term 1 FOLLOWER]: Advancing to term 2
I20250813 19:58:11.733464 12455 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fd83c2c55a04c8dadd09f4fdbc6c40f" candidate_uuid: "ce62b68d7392410fb8d42e3f3f11d552" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "dc47bb2a10f74261afb8ed36e751a73d"
I20250813 19:58:11.733879 12455 raft_consensus.cc:3053] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 1 LEADER]: Stepping down as leader of term 1
I20250813 19:58:11.734077 12455 raft_consensus.cc:738] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: dc47bb2a10f74261afb8ed36e751a73d, State: Running, Role: LEADER
I20250813 19:58:11.734530 12455 consensus_queue.cc:260] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
I20250813 19:58:11.735275 12455 raft_consensus.cc:3058] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 1 FOLLOWER]: Advancing to term 2
I20250813 19:58:11.736914 12721 raft_consensus.cc:2466] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce62b68d7392410fb8d42e3f3f11d552 in term 2.
I20250813 19:58:11.737674 12523 leader_election.cc:304] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3594f0d25e9f41a0ac9170f179db3dce, ce62b68d7392410fb8d42e3f3f11d552; no voters:
I20250813 19:58:11.739416 12835 raft_consensus.cc:2802] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 19:58:11.740084 12455 raft_consensus.cc:2466] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce62b68d7392410fb8d42e3f3f11d552 in term 2.
I20250813 19:58:11.740911 12835 raft_consensus.cc:695] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [term 2 LEADER]: Becoming Leader. State: Replica: ce62b68d7392410fb8d42e3f3f11d552, State: Running, Role: LEADER
I20250813 19:58:11.741757 12835 consensus_queue.cc:237] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } }
I20250813 19:58:11.748427 12312 catalog_manager.cc:5582] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 reported cstate change: term changed from 1 to 2, leader changed from dc47bb2a10f74261afb8ed36e751a73d (127.11.36.193) to ce62b68d7392410fb8d42e3f3f11d552 (127.11.36.194). New cstate: current_term: 2 leader_uuid: "ce62b68d7392410fb8d42e3f3f11d552" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ce62b68d7392410fb8d42e3f3f11d552" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 34549 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 } health_report { overall_health: UNKNOWN } } }
I20250813 19:58:12.151866 12455 raft_consensus.cc:1273] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 2 FOLLOWER]: Refusing update from remote peer ce62b68d7392410fb8d42e3f3f11d552: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250813 19:58:12.153113 12835 consensus_queue.cc:1035] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc47bb2a10f74261afb8ed36e751a73d" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 40577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250813 19:58:12.169494 12721 raft_consensus.cc:1273] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P 3594f0d25e9f41a0ac9170f179db3dce [term 2 FOLLOWER]: Refusing update from remote peer ce62b68d7392410fb8d42e3f3f11d552: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250813 19:58:12.170687 12835 consensus_queue.cc:1035] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P ce62b68d7392410fb8d42e3f3f11d552 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3594f0d25e9f41a0ac9170f179db3dce" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 41095 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250813 19:58:14.128450 12455 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "5fd83c2c55a04c8dadd09f4fdbc6c40f"
dest_uuid: "dc47bb2a10f74261afb8ed36e751a73d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54224
I20250813 19:58:14.129009 12455 raft_consensus.cc:604] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 2 FOLLOWER]: Received request to transfer leadership
I20250813 19:58:14.129316 12455 raft_consensus.cc:612] T 5fd83c2c55a04c8dadd09f4fdbc6c40f P dc47bb2a10f74261afb8ed36e751a73d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250813 19:58:15.213183 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 12371
I20250813 19:58:15.236603 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 12504
I20250813 19:58:15.260823 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 12637
I20250813 19:58:15.285526 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 12278
2025-08-13T19:58:15Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14402 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250813 19:58:15.340605 11411 test_util.cc:276] Using random seed: -1815333342
I20250813 19:58:15.344391 11411 ts_itest-base.cc:115] Starting cluster with:
I20250813 19:58:15.344539 11411 ts_itest-base.cc:116] --------------
I20250813 19:58:15.344686 11411 ts_itest-base.cc:117] 3 tablet servers
I20250813 19:58:15.344830 11411 ts_itest-base.cc:118] 3 replicas per TS
I20250813 19:58:15.344971 11411 ts_itest-base.cc:119] --------------
2025-08-13T19:58:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T19:58:15Z Disabled control of system clock
I20250813 19:58:15.377780 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:34137
--webserver_interface=127.11.36.254
--webserver_port=0
--builtin_ntp_servers=127.11.36.212:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:34137 with env {}
W20250813 19:58:15.659668 12878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:15.660234 12878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:15.660799 12878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:15.690642 12878 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 19:58:15.690907 12878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:15.691102 12878 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 19:58:15.691291 12878 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 19:58:15.723997 12878 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:34137
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:34137
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:15.725391 12878 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:15.726887 12878 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:15.737540 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:15.737838 12885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:16.874619 12887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:16.876593 12886 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1134 milliseconds
I20250813 19:58:16.876720 12878 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:16.877851 12878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:16.880774 12878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:16.882208 12878 hybrid_clock.cc:648] HybridClock initialized: now 1755115096882165 us; error 58 us; skew 500 ppm
I20250813 19:58:16.882937 12878 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:16.889137 12878 webserver.cc:480] Webserver started at http://127.11.36.254:39111/ using document root <none> and password file <none>
I20250813 19:58:16.889966 12878 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:16.890163 12878 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:16.890620 12878 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:16.896193 12878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4bbf91d635004d2f80d2ef0a7d360b66"
format_stamp: "Formatted at 2025-08-13 19:58:16 on dist-test-slave-l098"
I20250813 19:58:16.897344 12878 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4bbf91d635004d2f80d2ef0a7d360b66"
format_stamp: "Formatted at 2025-08-13 19:58:16 on dist-test-slave-l098"
I20250813 19:58:16.904003 12878 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250813 19:58:16.909200 12894 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:16.910142 12878 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.005s
I20250813 19:58:16.910466 12878 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4bbf91d635004d2f80d2ef0a7d360b66"
format_stamp: "Formatted at 2025-08-13 19:58:16 on dist-test-slave-l098"
I20250813 19:58:16.910758 12878 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:16.954550 12878 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:16.955862 12878 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:16.956284 12878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:17.020025 12878 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:34137
I20250813 19:58:17.020123 12945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:34137 every 8 connection(s)
I20250813 19:58:17.022449 12878 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 19:58:17.027046 12946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:17.028784 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 12878
I20250813 19:58:17.029232 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 19:58:17.046223 12946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66: Bootstrap starting.
I20250813 19:58:17.051777 12946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:17.053229 12946 log.cc:826] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:17.057124 12946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66: No bootstrap required, opened a new log
I20250813 19:58:17.072932 12946 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbf91d635004d2f80d2ef0a7d360b66" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 34137 } }
I20250813 19:58:17.073508 12946 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:17.073722 12946 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bbf91d635004d2f80d2ef0a7d360b66, State: Initialized, Role: FOLLOWER
I20250813 19:58:17.074326 12946 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbf91d635004d2f80d2ef0a7d360b66" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 34137 } }
I20250813 19:58:17.074748 12946 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:58:17.074996 12946 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:58:17.075259 12946 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:17.079316 12946 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbf91d635004d2f80d2ef0a7d360b66" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 34137 } }
I20250813 19:58:17.079982 12946 leader_election.cc:304] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bbf91d635004d2f80d2ef0a7d360b66; no voters:
I20250813 19:58:17.081691 12946 leader_election.cc:290] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 19:58:17.082614 12951 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:17.084967 12951 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [term 1 LEADER]: Becoming Leader. State: Replica: 4bbf91d635004d2f80d2ef0a7d360b66, State: Running, Role: LEADER
I20250813 19:58:17.085429 12946 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 19:58:17.085788 12951 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbf91d635004d2f80d2ef0a7d360b66" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 34137 } }
I20250813 19:58:17.096773 12953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bbf91d635004d2f80d2ef0a7d360b66. Latest consensus state: current_term: 1 leader_uuid: "4bbf91d635004d2f80d2ef0a7d360b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbf91d635004d2f80d2ef0a7d360b66" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 34137 } } }
I20250813 19:58:17.097100 12952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bbf91d635004d2f80d2ef0a7d360b66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bbf91d635004d2f80d2ef0a7d360b66" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 34137 } } }
I20250813 19:58:17.097388 12953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [sys.catalog]: This master's current role is: LEADER
I20250813 19:58:17.097581 12952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66 [sys.catalog]: This master's current role is: LEADER
I20250813 19:58:17.100030 12960 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 19:58:17.110316 12960 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 19:58:17.125967 12960 catalog_manager.cc:1349] Generated new cluster ID: 6151dcd558064019b988a702eef4f479
I20250813 19:58:17.126395 12960 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 19:58:17.155068 12960 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 19:58:17.156419 12960 catalog_manager.cc:1506] Loading token signing keys...
I20250813 19:58:17.170985 12960 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4bbf91d635004d2f80d2ef0a7d360b66: Generated new TSK 0
I20250813 19:58:17.171819 12960 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 19:58:17.188978 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:0
--local_ip_for_outbound_sockets=127.11.36.193
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:34137
--builtin_ntp_servers=127.11.36.212:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250813 19:58:17.462934 12970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:17.463373 12970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:17.463814 12970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:17.491843 12970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:17.492614 12970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 19:58:17.523449 12970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:34137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:17.524704 12970 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:17.526142 12970 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:17.537235 12976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:17.539873 12977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:18.937561 12979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:18.939313 12978 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1396 milliseconds
I20250813 19:58:18.939445 12970 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:18.940601 12970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:18.942679 12970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:18.944046 12970 hybrid_clock.cc:648] HybridClock initialized: now 1755115098944018 us; error 25 us; skew 500 ppm
I20250813 19:58:18.944747 12970 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:18.950244 12970 webserver.cc:480] Webserver started at http://127.11.36.193:33327/ using document root <none> and password file <none>
I20250813 19:58:18.951099 12970 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:18.951292 12970 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:18.951704 12970 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:18.955541 12970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "33806df1016e4f019c61e38b920aa032"
format_stamp: "Formatted at 2025-08-13 19:58:18 on dist-test-slave-l098"
I20250813 19:58:18.956554 12970 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "33806df1016e4f019c61e38b920aa032"
format_stamp: "Formatted at 2025-08-13 19:58:18 on dist-test-slave-l098"
I20250813 19:58:18.963200 12970 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250813 19:58:18.968999 12986 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:18.969903 12970 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250813 19:58:18.970199 12970 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "33806df1016e4f019c61e38b920aa032"
format_stamp: "Formatted at 2025-08-13 19:58:18 on dist-test-slave-l098"
I20250813 19:58:18.970511 12970 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:19.022378 12970 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:19.024026 12970 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:19.024497 12970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:19.027482 12970 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:19.032346 12970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:19.032533 12970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:19.032711 12970 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:19.032832 12970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:19.159843 12970 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:45073
I20250813 19:58:19.159927 13098 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:45073 every 8 connection(s)
I20250813 19:58:19.162236 12970 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 19:58:19.168247 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 12970
I20250813 19:58:19.168655 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 19:58:19.174367 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.194:0
--local_ip_for_outbound_sockets=127.11.36.194
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:34137
--builtin_ntp_servers=127.11.36.212:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:58:19.184854 13099 heartbeater.cc:344] Connected to a master server at 127.11.36.254:34137
I20250813 19:58:19.185253 13099 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:19.186173 13099 heartbeater.cc:507] Master 127.11.36.254:34137 requested a full tablet report, sending...
I20250813 19:58:19.188899 12911 ts_manager.cc:194] Registered new tserver with Master: 33806df1016e4f019c61e38b920aa032 (127.11.36.193:45073)
I20250813 19:58:19.190899 12911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:36163
W20250813 19:58:19.444482 13103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:19.445004 13103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:19.445470 13103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:19.473685 13103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:19.474502 13103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.194
I20250813 19:58:19.505664 13103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:34137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.194
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:19.506927 13103 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:19.508395 13103 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:19.518990 13109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:58:20.194280 13099 heartbeater.cc:499] Master 127.11.36.254:34137 was elected leader, sending a full tablet report...
W20250813 19:58:19.520561 13110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:20.611847 13112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:20.613829 13111 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1089 milliseconds
I20250813 19:58:20.613919 13103 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:20.615003 13103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:20.617435 13103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:20.618832 13103 hybrid_clock.cc:648] HybridClock initialized: now 1755115100618799 us; error 39 us; skew 500 ppm
I20250813 19:58:20.619580 13103 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:20.625669 13103 webserver.cc:480] Webserver started at http://127.11.36.194:42567/ using document root <none> and password file <none>
I20250813 19:58:20.626523 13103 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:20.626717 13103 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:20.627164 13103 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:20.631317 13103 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "950c4c52e6d24d868b8c8963041bb54d"
format_stamp: "Formatted at 2025-08-13 19:58:20 on dist-test-slave-l098"
I20250813 19:58:20.632355 13103 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "950c4c52e6d24d868b8c8963041bb54d"
format_stamp: "Formatted at 2025-08-13 19:58:20 on dist-test-slave-l098"
I20250813 19:58:20.638512 13103 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250813 19:58:20.643524 13119 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:20.644450 13103 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250813 19:58:20.644748 13103 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "950c4c52e6d24d868b8c8963041bb54d"
format_stamp: "Formatted at 2025-08-13 19:58:20 on dist-test-slave-l098"
I20250813 19:58:20.645030 13103 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:20.694304 13103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:20.695641 13103 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:20.696033 13103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:20.698246 13103 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:20.702127 13103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:20.702322 13103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:20.702526 13103 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:20.702661 13103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:20.823393 13103 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.194:42431
I20250813 19:58:20.823485 13231 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.194:42431 every 8 connection(s)
I20250813 19:58:20.825932 13103 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 19:58:20.831892 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 13103
I20250813 19:58:20.832286 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 19:58:20.837797 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.195:0
--local_ip_for_outbound_sockets=127.11.36.195
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:34137
--builtin_ntp_servers=127.11.36.212:35865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:58:20.846057 13232 heartbeater.cc:344] Connected to a master server at 127.11.36.254:34137
I20250813 19:58:20.846454 13232 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:20.847383 13232 heartbeater.cc:507] Master 127.11.36.254:34137 requested a full tablet report, sending...
I20250813 19:58:20.849393 12911 ts_manager.cc:194] Registered new tserver with Master: 950c4c52e6d24d868b8c8963041bb54d (127.11.36.194:42431)
I20250813 19:58:20.850564 12911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.194:57465
W20250813 19:58:21.126840 13236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:21.127517 13236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:21.128036 13236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:21.157119 13236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:21.158417 13236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.195
I20250813 19:58:21.196213 13236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:34137
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.195
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:21.197383 13236 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:21.198783 13236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:21.209906 13242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:58:21.853345 13232 heartbeater.cc:499] Master 127.11.36.254:34137 was elected leader, sending a full tablet report...
W20250813 19:58:21.210613 13243 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:22.583024 13245 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:22.585860 13244 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1370 milliseconds
W20250813 19:58:22.586048 13236 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.376s user 0.435s sys 0.919s
W20250813 19:58:22.586372 13236 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.376s user 0.435s sys 0.919s
I20250813 19:58:22.586632 13236 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:22.587635 13236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:22.589900 13236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:22.591246 13236 hybrid_clock.cc:648] HybridClock initialized: now 1755115102591218 us; error 38 us; skew 500 ppm
I20250813 19:58:22.591979 13236 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:22.598614 13236 webserver.cc:480] Webserver started at http://127.11.36.195:35821/ using document root <none> and password file <none>
I20250813 19:58:22.599584 13236 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:22.599828 13236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:22.600294 13236 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:22.604579 13236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a90843c434574d2aba9ed4529cd33744"
format_stamp: "Formatted at 2025-08-13 19:58:22 on dist-test-slave-l098"
I20250813 19:58:22.605610 13236 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a90843c434574d2aba9ed4529cd33744"
format_stamp: "Formatted at 2025-08-13 19:58:22 on dist-test-slave-l098"
I20250813 19:58:22.612829 13236 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.006s
I20250813 19:58:22.618367 13253 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:22.619451 13236 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250813 19:58:22.619745 13236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a90843c434574d2aba9ed4529cd33744"
format_stamp: "Formatted at 2025-08-13 19:58:22 on dist-test-slave-l098"
I20250813 19:58:22.620091 13236 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:22.692272 13236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:22.693641 13236 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:22.694082 13236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:22.696390 13236 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:22.700169 13236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:22.700368 13236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:22.700598 13236 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:22.700755 13236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:22.825479 13236 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.195:43099
I20250813 19:58:22.825527 13365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.195:43099 every 8 connection(s)
I20250813 19:58:22.829164 13236 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 19:58:22.838865 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 13236
I20250813 19:58:22.839566 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 19:58:22.852037 13366 heartbeater.cc:344] Connected to a master server at 127.11.36.254:34137
I20250813 19:58:22.852504 13366 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:22.853449 13366 heartbeater.cc:507] Master 127.11.36.254:34137 requested a full tablet report, sending...
I20250813 19:58:22.855289 12911 ts_manager.cc:194] Registered new tserver with Master: a90843c434574d2aba9ed4529cd33744 (127.11.36.195:43099)
I20250813 19:58:22.856391 12911 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.195:48727
I20250813 19:58:22.859948 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 19:58:22.891168 12910 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47506:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250813 19:58:22.909216 12910 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 19:58:22.958261 13034 tablet_service.cc:1468] Processing CreateTablet for tablet 5d7692c862b14b7482c977256108545f (DEFAULT_TABLE table=TestTable [id=de94787e83da4ea6a96d1f5db3df798a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:22.960345 13034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d7692c862b14b7482c977256108545f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:22.967555 13167 tablet_service.cc:1468] Processing CreateTablet for tablet 5d7692c862b14b7482c977256108545f (DEFAULT_TABLE table=TestTable [id=de94787e83da4ea6a96d1f5db3df798a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:22.969218 13167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d7692c862b14b7482c977256108545f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:22.972213 13301 tablet_service.cc:1468] Processing CreateTablet for tablet 5d7692c862b14b7482c977256108545f (DEFAULT_TABLE table=TestTable [id=de94787e83da4ea6a96d1f5db3df798a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:22.974002 13301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d7692c862b14b7482c977256108545f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:22.983670 13385 tablet_bootstrap.cc:492] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032: Bootstrap starting.
I20250813 19:58:22.989132 13385 tablet_bootstrap.cc:654] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:22.991290 13385 log.cc:826] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:22.995782 13387 tablet_bootstrap.cc:492] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d: Bootstrap starting.
I20250813 19:58:22.996670 13385 tablet_bootstrap.cc:492] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032: No bootstrap required, opened a new log
I20250813 19:58:22.997076 13385 ts_tablet_manager.cc:1397] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.005s
I20250813 19:58:23.002446 13388 tablet_bootstrap.cc:492] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744: Bootstrap starting.
I20250813 19:58:23.003528 13387 tablet_bootstrap.cc:654] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:23.005326 13387 log.cc:826] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:23.009557 13388 tablet_bootstrap.cc:654] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:23.009735 13387 tablet_bootstrap.cc:492] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d: No bootstrap required, opened a new log
I20250813 19:58:23.010154 13387 ts_tablet_manager.cc:1397] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20250813 19:58:23.012190 13388 log.cc:826] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:23.017293 13388 tablet_bootstrap.cc:492] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744: No bootstrap required, opened a new log
I20250813 19:58:23.017760 13388 ts_tablet_manager.cc:1397] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250813 19:58:23.021206 13385 raft_consensus.cc:357] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } }
I20250813 19:58:23.022198 13385 raft_consensus.cc:383] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:23.022521 13385 raft_consensus.cc:738] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33806df1016e4f019c61e38b920aa032, State: Initialized, Role: FOLLOWER
I20250813 19:58:23.023495 13385 consensus_queue.cc:260] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } }
I20250813 19:58:23.033535 13385 ts_tablet_manager.cc:1428] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032: Time spent starting tablet: real 0.036s user 0.024s sys 0.010s
I20250813 19:58:23.031491 13387 raft_consensus.cc:357] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } }
I20250813 19:58:23.035378 13387 raft_consensus.cc:383] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:23.035692 13387 raft_consensus.cc:738] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950c4c52e6d24d868b8c8963041bb54d, State: Initialized, Role: FOLLOWER
I20250813 19:58:23.036624 13387 consensus_queue.cc:260] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } }
I20250813 19:58:23.045780 13388 raft_consensus.cc:357] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } }
I20250813 19:58:23.046540 13388 raft_consensus.cc:383] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:23.046785 13388 raft_consensus.cc:738] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90843c434574d2aba9ed4529cd33744, State: Initialized, Role: FOLLOWER
I20250813 19:58:23.047199 13387 ts_tablet_manager.cc:1428] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d: Time spent starting tablet: real 0.037s user 0.022s sys 0.011s
I20250813 19:58:23.047470 13388 consensus_queue.cc:260] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } }
I20250813 19:58:23.050776 13366 heartbeater.cc:499] Master 127.11.36.254:34137 was elected leader, sending a full tablet report...
I20250813 19:58:23.051987 13388 ts_tablet_manager.cc:1428] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744: Time spent starting tablet: real 0.034s user 0.023s sys 0.008s
I20250813 19:58:23.055497 13392 raft_consensus.cc:491] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 19:58:23.055984 13392 raft_consensus.cc:513] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } }
I20250813 19:58:23.058496 13392 leader_election.cc:290] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33806df1016e4f019c61e38b920aa032 (127.11.36.193:45073), a90843c434574d2aba9ed4529cd33744 (127.11.36.195:43099)
I20250813 19:58:23.072888 13321 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d7692c862b14b7482c977256108545f" candidate_uuid: "950c4c52e6d24d868b8c8963041bb54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90843c434574d2aba9ed4529cd33744" is_pre_election: true
I20250813 19:58:23.073678 13321 raft_consensus.cc:2466] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 950c4c52e6d24d868b8c8963041bb54d in term 0.
I20250813 19:58:23.073890 13054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d7692c862b14b7482c977256108545f" candidate_uuid: "950c4c52e6d24d868b8c8963041bb54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33806df1016e4f019c61e38b920aa032" is_pre_election: true
I20250813 19:58:23.074695 13054 raft_consensus.cc:2466] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 950c4c52e6d24d868b8c8963041bb54d in term 0.
I20250813 19:58:23.075069 13121 leader_election.cc:304] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 950c4c52e6d24d868b8c8963041bb54d, a90843c434574d2aba9ed4529cd33744; no voters:
I20250813 19:58:23.075914 13392 raft_consensus.cc:2802] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 19:58:23.076189 13392 raft_consensus.cc:491] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 19:58:23.076445 13392 raft_consensus.cc:3058] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:23.081027 13392 raft_consensus.cc:513] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } }
W20250813 19:58:23.082397 13233 tablet.cc:2378] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:58:23.083138 13054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d7692c862b14b7482c977256108545f" candidate_uuid: "950c4c52e6d24d868b8c8963041bb54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33806df1016e4f019c61e38b920aa032"
I20250813 19:58:23.083540 13392 leader_election.cc:290] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [CANDIDATE]: Term 1 election: Requested vote from peers 33806df1016e4f019c61e38b920aa032 (127.11.36.193:45073), a90843c434574d2aba9ed4529cd33744 (127.11.36.195:43099)
I20250813 19:58:23.083609 13054 raft_consensus.cc:3058] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032 [term 0 FOLLOWER]: Advancing to term 1
W20250813 19:58:23.084228 13367 tablet.cc:2378] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:58:23.084266 13321 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5d7692c862b14b7482c977256108545f" candidate_uuid: "950c4c52e6d24d868b8c8963041bb54d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a90843c434574d2aba9ed4529cd33744"
I20250813 19:58:23.084813 13321 raft_consensus.cc:3058] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:23.088402 13054 raft_consensus.cc:2466] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 950c4c52e6d24d868b8c8963041bb54d in term 1.
I20250813 19:58:23.089229 13123 leader_election.cc:304] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33806df1016e4f019c61e38b920aa032, 950c4c52e6d24d868b8c8963041bb54d; no voters:
I20250813 19:58:23.089928 13392 raft_consensus.cc:2802] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:23.090787 13321 raft_consensus.cc:2466] T 5d7692c862b14b7482c977256108545f P a90843c434574d2aba9ed4529cd33744 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 950c4c52e6d24d868b8c8963041bb54d in term 1.
I20250813 19:58:23.092537 13392 raft_consensus.cc:695] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [term 1 LEADER]: Becoming Leader. State: Replica: 950c4c52e6d24d868b8c8963041bb54d, State: Running, Role: LEADER
I20250813 19:58:23.093348 13392 consensus_queue.cc:237] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } }
I20250813 19:58:23.104272 12911 catalog_manager.cc:5582] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d reported cstate change: term changed from 0 to 1, leader changed from <none> to 950c4c52e6d24d868b8c8963041bb54d (127.11.36.194). New cstate: current_term: 1 leader_uuid: "950c4c52e6d24d868b8c8963041bb54d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } health_report { overall_health: HEALTHY } } }
I20250813 19:58:23.130040 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 19:58:23.133314 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 33806df1016e4f019c61e38b920aa032 to finish bootstrapping
I20250813 19:58:23.145488 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 950c4c52e6d24d868b8c8963041bb54d to finish bootstrapping
I20250813 19:58:23.156111 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a90843c434574d2aba9ed4529cd33744 to finish bootstrapping
W20250813 19:58:23.170679 13100 tablet.cc:2378] T 5d7692c862b14b7482c977256108545f P 33806df1016e4f019c61e38b920aa032: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:58:23.170735 12911 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47506:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250813 19:58:23.172225 12911 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 19:58:23.189260 13034 tablet_service.cc:1468] Processing CreateTablet for tablet 8c41b4e6e5ae4eb781bc031262f7fd1d (DEFAULT_TABLE table=TestAnotherTable [id=eb66d23b6b8146e791cba8adadfa02f2]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250813 19:58:23.190325 13034 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c41b4e6e5ae4eb781bc031262f7fd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:23.190268 13167 tablet_service.cc:1468] Processing CreateTablet for tablet 8c41b4e6e5ae4eb781bc031262f7fd1d (DEFAULT_TABLE table=TestAnotherTable [id=eb66d23b6b8146e791cba8adadfa02f2]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250813 19:58:23.190505 13301 tablet_service.cc:1468] Processing CreateTablet for tablet 8c41b4e6e5ae4eb781bc031262f7fd1d (DEFAULT_TABLE table=TestAnotherTable [id=eb66d23b6b8146e791cba8adadfa02f2]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250813 19:58:23.191252 13301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c41b4e6e5ae4eb781bc031262f7fd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:23.191270 13167 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c41b4e6e5ae4eb781bc031262f7fd1d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:23.202957 13388 tablet_bootstrap.cc:492] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744: Bootstrap starting.
I20250813 19:58:23.205993 13385 tablet_bootstrap.cc:492] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032: Bootstrap starting.
I20250813 19:58:23.208443 13388 tablet_bootstrap.cc:654] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:23.210551 13387 tablet_bootstrap.cc:492] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d: Bootstrap starting.
I20250813 19:58:23.210592 13385 tablet_bootstrap.cc:654] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:23.215327 13388 tablet_bootstrap.cc:492] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744: No bootstrap required, opened a new log
I20250813 19:58:23.215745 13388 ts_tablet_manager.cc:1397] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744: Time spent bootstrapping tablet: real 0.013s user 0.012s sys 0.000s
I20250813 19:58:23.217371 13387 tablet_bootstrap.cc:654] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:23.218289 13388 raft_consensus.cc:357] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } }
I20250813 19:58:23.218984 13388 raft_consensus.cc:383] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:23.219240 13388 raft_consensus.cc:738] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a90843c434574d2aba9ed4529cd33744, State: Initialized, Role: FOLLOWER
I20250813 19:58:23.219941 13388 consensus_queue.cc:260] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } }
I20250813 19:58:23.221196 13385 tablet_bootstrap.cc:492] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032: No bootstrap required, opened a new log
I20250813 19:58:23.221611 13385 ts_tablet_manager.cc:1397] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20250813 19:58:23.222854 13387 tablet_bootstrap.cc:492] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d: No bootstrap required, opened a new log
I20250813 19:58:23.223270 13387 ts_tablet_manager.cc:1397] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.004s
I20250813 19:58:23.223613 13388 ts_tablet_manager.cc:1428] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20250813 19:58:23.224798 13385 raft_consensus.cc:357] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } }
I20250813 19:58:23.225471 13385 raft_consensus.cc:383] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:23.225771 13385 raft_consensus.cc:738] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33806df1016e4f019c61e38b920aa032, State: Initialized, Role: FOLLOWER
I20250813 19:58:23.225751 13387 raft_consensus.cc:357] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } }
I20250813 19:58:23.226305 13387 raft_consensus.cc:383] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:23.226616 13387 raft_consensus.cc:738] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 950c4c52e6d24d868b8c8963041bb54d, State: Initialized, Role: FOLLOWER
I20250813 19:58:23.226364 13385 consensus_queue.cc:260] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } }
I20250813 19:58:23.227216 13387 consensus_queue.cc:260] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } }
I20250813 19:58:23.228292 13385 ts_tablet_manager.cc:1428] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250813 19:58:23.229496 13387 ts_tablet_manager.cc:1428] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20250813 19:58:23.272392 13393 raft_consensus.cc:491] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 19:58:23.272760 13393 raft_consensus.cc:513] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } }
I20250813 19:58:23.275390 13393 leader_election.cc:290] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33806df1016e4f019c61e38b920aa032 (127.11.36.193:45073), 950c4c52e6d24d868b8c8963041bb54d (127.11.36.194:42431)
I20250813 19:58:23.284190 13054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c41b4e6e5ae4eb781bc031262f7fd1d" candidate_uuid: "a90843c434574d2aba9ed4529cd33744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33806df1016e4f019c61e38b920aa032" is_pre_election: true
I20250813 19:58:23.284693 13054 raft_consensus.cc:2466] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a90843c434574d2aba9ed4529cd33744 in term 0.
I20250813 19:58:23.285684 13257 leader_election.cc:304] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33806df1016e4f019c61e38b920aa032, a90843c434574d2aba9ed4529cd33744; no voters:
I20250813 19:58:23.286417 13393 raft_consensus.cc:2802] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 19:58:23.286691 13393 raft_consensus.cc:491] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 19:58:23.286564 13187 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c41b4e6e5ae4eb781bc031262f7fd1d" candidate_uuid: "a90843c434574d2aba9ed4529cd33744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "950c4c52e6d24d868b8c8963041bb54d" is_pre_election: true
I20250813 19:58:23.286931 13393 raft_consensus.cc:3058] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:23.287191 13187 raft_consensus.cc:2466] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a90843c434574d2aba9ed4529cd33744 in term 0.
I20250813 19:58:23.291533 13393 raft_consensus.cc:513] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } }
I20250813 19:58:23.292948 13393 leader_election.cc:290] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [CANDIDATE]: Term 1 election: Requested vote from peers 33806df1016e4f019c61e38b920aa032 (127.11.36.193:45073), 950c4c52e6d24d868b8c8963041bb54d (127.11.36.194:42431)
I20250813 19:58:23.293570 13054 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c41b4e6e5ae4eb781bc031262f7fd1d" candidate_uuid: "a90843c434574d2aba9ed4529cd33744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33806df1016e4f019c61e38b920aa032"
I20250813 19:58:23.293813 13187 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8c41b4e6e5ae4eb781bc031262f7fd1d" candidate_uuid: "a90843c434574d2aba9ed4529cd33744" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "950c4c52e6d24d868b8c8963041bb54d"
I20250813 19:58:23.294100 13054 raft_consensus.cc:3058] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:23.294312 13187 raft_consensus.cc:3058] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:23.300555 13054 raft_consensus.cc:2466] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 33806df1016e4f019c61e38b920aa032 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a90843c434574d2aba9ed4529cd33744 in term 1.
I20250813 19:58:23.300788 13187 raft_consensus.cc:2466] T 8c41b4e6e5ae4eb781bc031262f7fd1d P 950c4c52e6d24d868b8c8963041bb54d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a90843c434574d2aba9ed4529cd33744 in term 1.
I20250813 19:58:23.301532 13257 leader_election.cc:304] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33806df1016e4f019c61e38b920aa032, a90843c434574d2aba9ed4529cd33744; no voters:
I20250813 19:58:23.302089 13393 raft_consensus.cc:2802] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:23.303584 13393 raft_consensus.cc:695] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [term 1 LEADER]: Becoming Leader. State: Replica: a90843c434574d2aba9ed4529cd33744, State: Running, Role: LEADER
I20250813 19:58:23.304262 13393 consensus_queue.cc:237] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } }
I20250813 19:58:23.313866 12910 catalog_manager.cc:5582] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 reported cstate change: term changed from 0 to 1, leader changed from <none> to a90843c434574d2aba9ed4529cd33744 (127.11.36.195). New cstate: current_term: 1 leader_uuid: "a90843c434574d2aba9ed4529cd33744" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 } health_report { overall_health: HEALTHY } } }
I20250813 19:58:23.536552 13392 consensus_queue.cc:1035] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [LEADER]: Connected to new peer: Peer: permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 19:58:23.554693 13409 consensus_queue.cc:1035] T 5d7692c862b14b7482c977256108545f P 950c4c52e6d24d868b8c8963041bb54d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a90843c434574d2aba9ed4529cd33744" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250813 19:58:23.634600 13405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:23.635170 13405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:23.662331 13405 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250813 19:58:23.754576 13393 consensus_queue.cc:1035] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33806df1016e4f019c61e38b920aa032" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 45073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 19:58:23.769851 13393 consensus_queue.cc:1035] T 8c41b4e6e5ae4eb781bc031262f7fd1d P a90843c434574d2aba9ed4529cd33744 [LEADER]: Connected to new peer: Peer: permanent_uuid: "950c4c52e6d24d868b8c8963041bb54d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250813 19:58:24.961697 13405 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.263s user 0.430s sys 0.820s
W20250813 19:58:24.962093 13405 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.263s user 0.430s sys 0.820s
W20250813 19:58:26.320215 13432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:26.320797 13432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:26.355108 13432 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250813 19:58:27.573284 13432 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.172s user 0.434s sys 0.737s
W20250813 19:58:27.573655 13432 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.173s user 0.434s sys 0.737s
W20250813 19:58:28.941242 13448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:28.941792 13448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:28.970358 13448 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250813 19:58:30.147215 13448 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.136s user 0.412s sys 0.722s
W20250813 19:58:30.147656 13448 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.137s user 0.413s sys 0.724s
W20250813 19:58:31.498267 13463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:31.498780 13463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:31.527468 13463 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250813 19:58:32.746459 13463 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.182s user 0.466s sys 0.713s
W20250813 19:58:32.746757 13463 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.182s user 0.466s sys 0.713s
I20250813 19:58:33.813977 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 12970
I20250813 19:58:33.842628 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 13103
I20250813 19:58:33.869122 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 13236
I20250813 19:58:33.894098 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 12878
2025-08-13T19:58:33Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18612 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250813 19:58:33.953704 11411 test_util.cc:276] Using random seed: -1796720241
I20250813 19:58:33.957862 11411 ts_itest-base.cc:115] Starting cluster with:
I20250813 19:58:33.958043 11411 ts_itest-base.cc:116] --------------
I20250813 19:58:33.958199 11411 ts_itest-base.cc:117] 3 tablet servers
I20250813 19:58:33.958330 11411 ts_itest-base.cc:118] 3 replicas per TS
I20250813 19:58:33.958463 11411 ts_itest-base.cc:119] --------------
2025-08-13T19:58:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T19:58:33Z Disabled control of system clock
I20250813 19:58:33.993649 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:36713
--webserver_interface=127.11.36.254
--webserver_port=0
--builtin_ntp_servers=127.11.36.212:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:36713
--superuser_acl=no-such-user with env {}
W20250813 19:58:34.275002 13488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:34.275552 13488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:34.276008 13488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:34.305505 13488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 19:58:34.305856 13488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:34.306110 13488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 19:58:34.306344 13488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 19:58:34.339352 13488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:36713
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:36713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:34.340616 13488 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:34.342108 13488 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:34.353214 13494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:34.353565 13495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:35.487514 13497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:35.489559 13496 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1129 milliseconds
I20250813 19:58:35.489635 13488 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:35.490801 13488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:35.493232 13488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:35.494554 13488 hybrid_clock.cc:648] HybridClock initialized: now 1755115115494525 us; error 30 us; skew 500 ppm
I20250813 19:58:35.495249 13488 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:35.500545 13488 webserver.cc:480] Webserver started at http://127.11.36.254:46009/ using document root <none> and password file <none>
I20250813 19:58:35.501390 13488 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:35.501586 13488 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:35.502010 13488 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:35.506130 13488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "36e98d13ab294c629cbba1ca5dd6af93"
format_stamp: "Formatted at 2025-08-13 19:58:35 on dist-test-slave-l098"
I20250813 19:58:35.507090 13488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "36e98d13ab294c629cbba1ca5dd6af93"
format_stamp: "Formatted at 2025-08-13 19:58:35 on dist-test-slave-l098"
I20250813 19:58:35.513545 13488 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250813 19:58:35.518358 13504 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:35.519217 13488 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250813 19:58:35.519488 13488 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "36e98d13ab294c629cbba1ca5dd6af93"
format_stamp: "Formatted at 2025-08-13 19:58:35 on dist-test-slave-l098"
I20250813 19:58:35.519785 13488 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:35.566965 13488 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:35.568394 13488 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:35.568800 13488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:35.636595 13488 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:36713
I20250813 19:58:35.636665 13555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:36713 every 8 connection(s)
I20250813 19:58:35.639220 13488 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 19:58:35.644039 13556 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:35.646281 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 13488
I20250813 19:58:35.646760 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 19:58:35.665230 13556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93: Bootstrap starting.
I20250813 19:58:35.670470 13556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:35.672442 13556 log.cc:826] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:35.677536 13556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93: No bootstrap required, opened a new log
I20250813 19:58:35.695325 13556 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e98d13ab294c629cbba1ca5dd6af93" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 36713 } }
I20250813 19:58:35.696024 13556 raft_consensus.cc:383] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:35.696352 13556 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36e98d13ab294c629cbba1ca5dd6af93, State: Initialized, Role: FOLLOWER
I20250813 19:58:35.697021 13556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e98d13ab294c629cbba1ca5dd6af93" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 36713 } }
I20250813 19:58:35.697566 13556 raft_consensus.cc:397] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:58:35.697934 13556 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:58:35.698302 13556 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:35.702585 13556 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e98d13ab294c629cbba1ca5dd6af93" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 36713 } }
I20250813 19:58:35.703346 13556 leader_election.cc:304] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36e98d13ab294c629cbba1ca5dd6af93; no voters:
I20250813 19:58:35.705039 13556 leader_election.cc:290] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 19:58:35.705767 13561 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:35.708333 13561 raft_consensus.cc:695] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [term 1 LEADER]: Becoming Leader. State: Replica: 36e98d13ab294c629cbba1ca5dd6af93, State: Running, Role: LEADER
I20250813 19:58:35.709079 13561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e98d13ab294c629cbba1ca5dd6af93" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 36713 } }
I20250813 19:58:35.709406 13556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 19:58:35.719045 13562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36e98d13ab294c629cbba1ca5dd6af93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e98d13ab294c629cbba1ca5dd6af93" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 36713 } } }
I20250813 19:58:35.719597 13563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36e98d13ab294c629cbba1ca5dd6af93. Latest consensus state: current_term: 1 leader_uuid: "36e98d13ab294c629cbba1ca5dd6af93" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36e98d13ab294c629cbba1ca5dd6af93" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 36713 } } }
I20250813 19:58:35.719905 13562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [sys.catalog]: This master's current role is: LEADER
I20250813 19:58:35.720219 13563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93 [sys.catalog]: This master's current role is: LEADER
I20250813 19:58:35.726204 13570 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 19:58:35.738991 13570 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 19:58:35.753494 13570 catalog_manager.cc:1349] Generated new cluster ID: 11258d6537244db6a94044393296c892
I20250813 19:58:35.753794 13570 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 19:58:35.784142 13570 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 19:58:35.785488 13570 catalog_manager.cc:1506] Loading token signing keys...
I20250813 19:58:35.796182 13570 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 36e98d13ab294c629cbba1ca5dd6af93: Generated new TSK 0
I20250813 19:58:35.797032 13570 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 19:58:35.810437 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:0
--local_ip_for_outbound_sockets=127.11.36.193
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:36713
--builtin_ntp_servers=127.11.36.212:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250813 19:58:36.088006 13580 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:36.088542 13580 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:36.089053 13580 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:36.117376 13580 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:36.118218 13580 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 19:58:36.149454 13580 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:36713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:36.150681 13580 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:36.152325 13580 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:36.165740 13587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:37.567576 13585 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 13580
W20250813 19:58:37.956817 13585 kernel_stack_watchdog.cc:198] Thread 13580 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250813 19:58:36.166847 13586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:37.959076 13589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:37.959585 13580 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.795s user 0.562s sys 1.220s
W20250813 19:58:37.959937 13580 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.795s user 0.562s sys 1.220s
W20250813 19:58:37.961810 13588 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1796 milliseconds
I20250813 19:58:37.961866 13580 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:37.962988 13580 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:37.965101 13580 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:37.966421 13580 hybrid_clock.cc:648] HybridClock initialized: now 1755115117966334 us; error 97 us; skew 500 ppm
I20250813 19:58:37.967089 13580 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:37.973250 13580 webserver.cc:480] Webserver started at http://127.11.36.193:33735/ using document root <none> and password file <none>
I20250813 19:58:37.974115 13580 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:37.974311 13580 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:37.974709 13580 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:37.978528 13580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "bc80d41966c24602857b561591a97eb6"
format_stamp: "Formatted at 2025-08-13 19:58:37 on dist-test-slave-l098"
I20250813 19:58:37.979460 13580 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "bc80d41966c24602857b561591a97eb6"
format_stamp: "Formatted at 2025-08-13 19:58:37 on dist-test-slave-l098"
I20250813 19:58:37.986387 13580 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250813 19:58:37.991423 13596 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:37.992444 13580 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250813 19:58:37.992738 13580 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "bc80d41966c24602857b561591a97eb6"
format_stamp: "Formatted at 2025-08-13 19:58:37 on dist-test-slave-l098"
I20250813 19:58:37.993094 13580 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:38.045104 13580 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:38.046381 13580 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:38.046770 13580 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:38.049242 13580 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:38.053056 13580 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:38.053261 13580 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:38.053512 13580 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:38.053671 13580 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:38.206831 13580 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:42671
I20250813 19:58:38.206992 13708 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:42671 every 8 connection(s)
I20250813 19:58:38.209237 13580 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 19:58:38.210654 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 13580
I20250813 19:58:38.211122 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 19:58:38.222324 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.194:0
--local_ip_for_outbound_sockets=127.11.36.194
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:36713
--builtin_ntp_servers=127.11.36.212:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:58:38.231742 13709 heartbeater.cc:344] Connected to a master server at 127.11.36.254:36713
I20250813 19:58:38.232246 13709 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:38.233546 13709 heartbeater.cc:507] Master 127.11.36.254:36713 requested a full tablet report, sending...
I20250813 19:58:38.236593 13521 ts_manager.cc:194] Registered new tserver with Master: bc80d41966c24602857b561591a97eb6 (127.11.36.193:42671)
I20250813 19:58:38.239460 13521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:57161
W20250813 19:58:38.505649 13713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:38.506112 13713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:38.506548 13713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:38.535213 13713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:38.536006 13713 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.194
I20250813 19:58:38.567406 13713 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:36713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.194
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:38.568692 13713 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:38.570114 13713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:38.580768 13719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:58:39.243645 13709 heartbeater.cc:499] Master 127.11.36.254:36713 was elected leader, sending a full tablet report...
W20250813 19:58:39.984578 13718 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 13713
W20250813 19:58:40.165985 13713 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.586s user 0.524s sys 1.055s
W20250813 19:58:38.581171 13720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:40.166357 13713 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.586s user 0.524s sys 1.055s
W20250813 19:58:40.168632 13722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:40.170912 13721 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1585 milliseconds
I20250813 19:58:40.170930 13713 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:40.171990 13713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:40.173934 13713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:40.175268 13713 hybrid_clock.cc:648] HybridClock initialized: now 1755115120175213 us; error 43 us; skew 500 ppm
I20250813 19:58:40.175932 13713 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:40.181586 13713 webserver.cc:480] Webserver started at http://127.11.36.194:45441/ using document root <none> and password file <none>
I20250813 19:58:40.182734 13713 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:40.183005 13713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:40.183498 13713 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:40.187526 13713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "46f7e7f0d9cd493db560e21f3a508514"
format_stamp: "Formatted at 2025-08-13 19:58:40 on dist-test-slave-l098"
I20250813 19:58:40.188524 13713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "46f7e7f0d9cd493db560e21f3a508514"
format_stamp: "Formatted at 2025-08-13 19:58:40 on dist-test-slave-l098"
I20250813 19:58:40.194990 13713 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250813 19:58:40.200008 13729 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:40.200976 13713 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250813 19:58:40.201272 13713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "46f7e7f0d9cd493db560e21f3a508514"
format_stamp: "Formatted at 2025-08-13 19:58:40 on dist-test-slave-l098"
I20250813 19:58:40.201570 13713 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:40.257793 13713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:40.259202 13713 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:40.259620 13713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:40.261943 13713 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:40.265729 13713 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:40.265914 13713 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:40.266132 13713 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:40.266284 13713 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:40.389425 13713 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.194:42661
I20250813 19:58:40.389503 13841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.194:42661 every 8 connection(s)
I20250813 19:58:40.391819 13713 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 19:58:40.399919 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 13713
I20250813 19:58:40.400336 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 19:58:40.405983 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.195:0
--local_ip_for_outbound_sockets=127.11.36.195
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:36713
--builtin_ntp_servers=127.11.36.212:35369
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:58:40.412214 13842 heartbeater.cc:344] Connected to a master server at 127.11.36.254:36713
I20250813 19:58:40.412629 13842 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:40.413609 13842 heartbeater.cc:507] Master 127.11.36.254:36713 requested a full tablet report, sending...
I20250813 19:58:40.415838 13521 ts_manager.cc:194] Registered new tserver with Master: 46f7e7f0d9cd493db560e21f3a508514 (127.11.36.194:42661)
I20250813 19:58:40.416960 13521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.194:57815
W20250813 19:58:40.690620 13846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:40.691093 13846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:40.691630 13846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:40.722065 13846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:40.722934 13846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.195
I20250813 19:58:40.756930 13846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:35369
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:36713
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.195
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:40.758265 13846 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:40.759878 13846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:40.770959 13852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:58:41.420337 13842 heartbeater.cc:499] Master 127.11.36.254:36713 was elected leader, sending a full tablet report...
W20250813 19:58:40.773201 13853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:41.904932 13854 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250813 19:58:41.906639 13855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:41.909108 13846 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.137s user 0.413s sys 0.722s
W20250813 19:58:41.909380 13846 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.138s user 0.413s sys 0.722s
I20250813 19:58:41.909605 13846 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:41.910688 13846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:41.916740 13846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:41.918170 13846 hybrid_clock.cc:648] HybridClock initialized: now 1755115121918142 us; error 47 us; skew 500 ppm
I20250813 19:58:41.918968 13846 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:41.925936 13846 webserver.cc:480] Webserver started at http://127.11.36.195:39729/ using document root <none> and password file <none>
I20250813 19:58:41.926826 13846 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:41.927011 13846 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:41.927446 13846 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:41.931583 13846 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "2877ebc34abe4e7baa44718e6044df2a"
format_stamp: "Formatted at 2025-08-13 19:58:41 on dist-test-slave-l098"
I20250813 19:58:41.932785 13846 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "2877ebc34abe4e7baa44718e6044df2a"
format_stamp: "Formatted at 2025-08-13 19:58:41 on dist-test-slave-l098"
I20250813 19:58:41.940017 13846 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250813 19:58:41.945755 13863 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:41.946874 13846 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250813 19:58:41.947171 13846 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "2877ebc34abe4e7baa44718e6044df2a"
format_stamp: "Formatted at 2025-08-13 19:58:41 on dist-test-slave-l098"
I20250813 19:58:41.947515 13846 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:42.019896 13846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:42.021297 13846 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:42.021704 13846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:42.024143 13846 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:42.028155 13846 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:42.028362 13846 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:42.028589 13846 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:42.028743 13846 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:42.158264 13846 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.195:44199
I20250813 19:58:42.158385 13975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.195:44199 every 8 connection(s)
I20250813 19:58:42.160715 13846 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 19:58:42.166759 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 13846
I20250813 19:58:42.167399 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 19:58:42.187988 13976 heartbeater.cc:344] Connected to a master server at 127.11.36.254:36713
I20250813 19:58:42.188411 13976 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:42.189280 13976 heartbeater.cc:507] Master 127.11.36.254:36713 requested a full tablet report, sending...
I20250813 19:58:42.191018 13521 ts_manager.cc:194] Registered new tserver with Master: 2877ebc34abe4e7baa44718e6044df2a (127.11.36.195:44199)
I20250813 19:58:42.192155 13521 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.195:52511
I20250813 19:58:42.199681 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 19:58:42.231571 13521 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43094:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250813 19:58:42.249470 13521 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 19:58:42.294696 13644 tablet_service.cc:1468] Processing CreateTablet for tablet d629281f1c2b44febd22129ba1386a75 (DEFAULT_TABLE table=TestTable [id=12fbad9db42f4a8896c5cf8ecd1001ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:42.296620 13644 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d629281f1c2b44febd22129ba1386a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:42.297255 13911 tablet_service.cc:1468] Processing CreateTablet for tablet d629281f1c2b44febd22129ba1386a75 (DEFAULT_TABLE table=TestTable [id=12fbad9db42f4a8896c5cf8ecd1001ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:42.297747 13777 tablet_service.cc:1468] Processing CreateTablet for tablet d629281f1c2b44febd22129ba1386a75 (DEFAULT_TABLE table=TestTable [id=12fbad9db42f4a8896c5cf8ecd1001ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:42.298645 13911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d629281f1c2b44febd22129ba1386a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:42.299335 13777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d629281f1c2b44febd22129ba1386a75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:42.322055 13995 tablet_bootstrap.cc:492] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a: Bootstrap starting.
I20250813 19:58:42.324204 13996 tablet_bootstrap.cc:492] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6: Bootstrap starting.
I20250813 19:58:42.326539 13997 tablet_bootstrap.cc:492] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514: Bootstrap starting.
I20250813 19:58:42.330243 13995 tablet_bootstrap.cc:654] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:42.331363 13996 tablet_bootstrap.cc:654] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:42.332368 13997 tablet_bootstrap.cc:654] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:42.333542 13996 log.cc:826] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:42.333917 13995 log.cc:826] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:42.334685 13997 log.cc:826] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:42.338112 13996 tablet_bootstrap.cc:492] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6: No bootstrap required, opened a new log
I20250813 19:58:42.338555 13996 ts_tablet_manager.cc:1397] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.005s
I20250813 19:58:42.340337 13997 tablet_bootstrap.cc:492] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514: No bootstrap required, opened a new log
I20250813 19:58:42.340842 13997 ts_tablet_manager.cc:1397] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.010s
I20250813 19:58:42.341202 13995 tablet_bootstrap.cc:492] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a: No bootstrap required, opened a new log
I20250813 19:58:42.341742 13995 ts_tablet_manager.cc:1397] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.011s
I20250813 19:58:42.357328 13996 raft_consensus.cc:357] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } }
I20250813 19:58:42.357965 13996 raft_consensus.cc:383] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:42.358244 13996 raft_consensus.cc:738] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc80d41966c24602857b561591a97eb6, State: Initialized, Role: FOLLOWER
I20250813 19:58:42.358978 13996 consensus_queue.cc:260] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } }
I20250813 19:58:42.362887 13996 ts_tablet_manager.cc:1428] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6: Time spent starting tablet: real 0.024s user 0.012s sys 0.010s
I20250813 19:58:42.367033 13997 raft_consensus.cc:357] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } }
I20250813 19:58:42.367702 13997 raft_consensus.cc:383] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:42.367960 13997 raft_consensus.cc:738] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46f7e7f0d9cd493db560e21f3a508514, State: Initialized, Role: FOLLOWER
I20250813 19:58:42.368165 13995 raft_consensus.cc:357] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } }
I20250813 19:58:42.369027 13995 raft_consensus.cc:383] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:42.368795 13997 consensus_queue.cc:260] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } }
I20250813 19:58:42.369308 13995 raft_consensus.cc:738] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2877ebc34abe4e7baa44718e6044df2a, State: Initialized, Role: FOLLOWER
I20250813 19:58:42.370155 13995 consensus_queue.cc:260] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } }
I20250813 19:58:42.373147 13997 ts_tablet_manager.cc:1428] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514: Time spent starting tablet: real 0.032s user 0.025s sys 0.007s
I20250813 19:58:42.375228 13976 heartbeater.cc:499] Master 127.11.36.254:36713 was elected leader, sending a full tablet report...
I20250813 19:58:42.375643 13995 ts_tablet_manager.cc:1428] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a: Time spent starting tablet: real 0.034s user 0.023s sys 0.009s
W20250813 19:58:42.398429 13843 tablet.cc:2378] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 19:58:42.414724 13977 tablet.cc:2378] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:58:42.463599 14001 raft_consensus.cc:491] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 19:58:42.464092 14001 raft_consensus.cc:513] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } }
I20250813 19:58:42.466384 14001 leader_election.cc:290] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46f7e7f0d9cd493db560e21f3a508514 (127.11.36.194:42661), 2877ebc34abe4e7baa44718e6044df2a (127.11.36.195:44199)
I20250813 19:58:42.468591 14002 raft_consensus.cc:491] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 19:58:42.469198 14002 raft_consensus.cc:513] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } }
W20250813 19:58:42.470220 13710 tablet.cc:2378] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:58:42.472159 14002 leader_election.cc:290] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc80d41966c24602857b561591a97eb6 (127.11.36.193:42671), 2877ebc34abe4e7baa44718e6044df2a (127.11.36.195:44199)
I20250813 19:58:42.483280 13797 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d629281f1c2b44febd22129ba1386a75" candidate_uuid: "bc80d41966c24602857b561591a97eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46f7e7f0d9cd493db560e21f3a508514" is_pre_election: true
I20250813 19:58:42.484095 13797 raft_consensus.cc:2466] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc80d41966c24602857b561591a97eb6 in term 0.
I20250813 19:58:42.485709 13600 leader_election.cc:304] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46f7e7f0d9cd493db560e21f3a508514, bc80d41966c24602857b561591a97eb6; no voters:
I20250813 19:58:42.486596 14001 raft_consensus.cc:2802] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 19:58:42.486554 13664 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d629281f1c2b44febd22129ba1386a75" candidate_uuid: "46f7e7f0d9cd493db560e21f3a508514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc80d41966c24602857b561591a97eb6" is_pre_election: true
I20250813 19:58:42.486922 14001 raft_consensus.cc:491] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 19:58:42.487215 14001 raft_consensus.cc:3058] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:42.489276 13931 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d629281f1c2b44febd22129ba1386a75" candidate_uuid: "bc80d41966c24602857b561591a97eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877ebc34abe4e7baa44718e6044df2a" is_pre_election: true
I20250813 19:58:42.490226 13931 raft_consensus.cc:2466] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bc80d41966c24602857b561591a97eb6 in term 0.
I20250813 19:58:42.493371 14001 raft_consensus.cc:513] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } }
I20250813 19:58:42.493790 13931 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d629281f1c2b44febd22129ba1386a75" candidate_uuid: "46f7e7f0d9cd493db560e21f3a508514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877ebc34abe4e7baa44718e6044df2a" is_pre_election: true
I20250813 19:58:42.494277 13931 raft_consensus.cc:2466] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46f7e7f0d9cd493db560e21f3a508514 in term 0.
I20250813 19:58:42.494426 13664 raft_consensus.cc:2391] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46f7e7f0d9cd493db560e21f3a508514 in current term 1: Already voted for candidate bc80d41966c24602857b561591a97eb6 in this term.
I20250813 19:58:42.495235 13730 leader_election.cc:304] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2877ebc34abe4e7baa44718e6044df2a, 46f7e7f0d9cd493db560e21f3a508514; no voters:
I20250813 19:58:42.495759 13797 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d629281f1c2b44febd22129ba1386a75" candidate_uuid: "bc80d41966c24602857b561591a97eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46f7e7f0d9cd493db560e21f3a508514"
I20250813 19:58:42.496186 14001 leader_election.cc:290] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [CANDIDATE]: Term 1 election: Requested vote from peers 46f7e7f0d9cd493db560e21f3a508514 (127.11.36.194:42661), 2877ebc34abe4e7baa44718e6044df2a (127.11.36.195:44199)
I20250813 19:58:42.496415 13797 raft_consensus.cc:3058] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:42.497143 13931 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d629281f1c2b44febd22129ba1386a75" candidate_uuid: "bc80d41966c24602857b561591a97eb6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2877ebc34abe4e7baa44718e6044df2a"
I20250813 19:58:42.497486 13931 raft_consensus.cc:3058] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:42.501590 13931 raft_consensus.cc:2466] T d629281f1c2b44febd22129ba1386a75 P 2877ebc34abe4e7baa44718e6044df2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc80d41966c24602857b561591a97eb6 in term 1.
I20250813 19:58:42.502251 13797 raft_consensus.cc:2466] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bc80d41966c24602857b561591a97eb6 in term 1.
I20250813 19:58:42.502519 13597 leader_election.cc:304] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2877ebc34abe4e7baa44718e6044df2a, bc80d41966c24602857b561591a97eb6; no voters:
I20250813 19:58:42.502722 14002 raft_consensus.cc:2762] T d629281f1c2b44febd22129ba1386a75 P 46f7e7f0d9cd493db560e21f3a508514 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250813 19:58:42.503290 14001 raft_consensus.cc:2802] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:42.505430 14001 raft_consensus.cc:695] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [term 1 LEADER]: Becoming Leader. State: Replica: bc80d41966c24602857b561591a97eb6, State: Running, Role: LEADER
I20250813 19:58:42.506202 14001 consensus_queue.cc:237] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } }
I20250813 19:58:42.514647 13520 catalog_manager.cc:5582] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 reported cstate change: term changed from 0 to 1, leader changed from <none> to bc80d41966c24602857b561591a97eb6 (127.11.36.193). New cstate: current_term: 1 leader_uuid: "bc80d41966c24602857b561591a97eb6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc80d41966c24602857b561591a97eb6" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 } health_report { overall_health: UNKNOWN } } }
I20250813 19:58:42.534847 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 19:58:42.537976 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver bc80d41966c24602857b561591a97eb6 to finish bootstrapping
I20250813 19:58:42.549830 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 46f7e7f0d9cd493db560e21f3a508514 to finish bootstrapping
I20250813 19:58:42.559484 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2877ebc34abe4e7baa44718e6044df2a to finish bootstrapping
I20250813 19:58:42.934356 14001 consensus_queue.cc:1035] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46f7e7f0d9cd493db560e21f3a508514" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 42661 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 19:58:42.954442 14001 consensus_queue.cc:1035] T d629281f1c2b44febd22129ba1386a75 P bc80d41966c24602857b561591a97eb6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2877ebc34abe4e7baa44718e6044df2a" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250813 19:58:44.120349 13520 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:47556
I20250813 19:58:45.148125 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 13580
I20250813 19:58:45.172708 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 13713
I20250813 19:58:45.195101 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 13846
I20250813 19:58:45.219437 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 13488
2025-08-13T19:58:45Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11315 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250813 19:58:45.269652 11411 test_util.cc:276] Using random seed: -1785404293
I20250813 19:58:45.273245 11411 ts_itest-base.cc:115] Starting cluster with:
I20250813 19:58:45.273437 11411 ts_itest-base.cc:116] --------------
I20250813 19:58:45.273597 11411 ts_itest-base.cc:117] 3 tablet servers
I20250813 19:58:45.273738 11411 ts_itest-base.cc:118] 3 replicas per TS
I20250813 19:58:45.273878 11411 ts_itest-base.cc:119] --------------
2025-08-13T19:58:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T19:58:45Z Disabled control of system clock
I20250813 19:58:45.307436 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:40459
--webserver_interface=127.11.36.254
--webserver_port=0
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:40459 with env {}
W20250813 19:58:45.588611 14048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:45.589187 14048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:45.589648 14048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:45.618629 14048 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 19:58:45.618898 14048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:45.619131 14048 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 19:58:45.619346 14048 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 19:58:45.650657 14048 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:40459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:45.651844 14048 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:45.653294 14048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:45.662272 14054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:47.066797 14053 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 14048
W20250813 19:58:47.339959 14048 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.677s user 0.631s sys 1.046s
W20250813 19:58:47.340420 14048 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.677s user 0.631s sys 1.046s
W20250813 19:58:45.663883 14055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:47.342206 14057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:47.345525 14056 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1678 milliseconds
I20250813 19:58:47.345595 14048 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:47.346731 14048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:47.349088 14048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:47.350442 14048 hybrid_clock.cc:648] HybridClock initialized: now 1755115127350422 us; error 66 us; skew 500 ppm
I20250813 19:58:47.351220 14048 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:47.356894 14048 webserver.cc:480] Webserver started at http://127.11.36.254:43741/ using document root <none> and password file <none>
I20250813 19:58:47.357772 14048 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:47.357961 14048 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:47.358390 14048 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:47.362278 14048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ecf90acd0b29462196a47a7b543701c8"
format_stamp: "Formatted at 2025-08-13 19:58:47 on dist-test-slave-l098"
I20250813 19:58:47.363328 14048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ecf90acd0b29462196a47a7b543701c8"
format_stamp: "Formatted at 2025-08-13 19:58:47 on dist-test-slave-l098"
I20250813 19:58:47.369935 14048 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.006s
I20250813 19:58:47.374934 14064 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:47.375867 14048 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250813 19:58:47.376168 14048 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ecf90acd0b29462196a47a7b543701c8"
format_stamp: "Formatted at 2025-08-13 19:58:47 on dist-test-slave-l098"
I20250813 19:58:47.376466 14048 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:47.431578 14048 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:47.433364 14048 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:47.433928 14048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:47.503566 14048 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:40459
I20250813 19:58:47.503654 14115 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:40459 every 8 connection(s)
I20250813 19:58:47.506124 14048 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 19:58:47.508038 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 14048
I20250813 19:58:47.508554 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 19:58:47.512157 14116 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:47.533710 14116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Bootstrap starting.
I20250813 19:58:47.538668 14116 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:47.540328 14116 log.cc:826] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:47.544442 14116 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: No bootstrap required, opened a new log
I20250813 19:58:47.562017 14116 raft_consensus.cc:357] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:58:47.562646 14116 raft_consensus.cc:383] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:47.562863 14116 raft_consensus.cc:738] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf90acd0b29462196a47a7b543701c8, State: Initialized, Role: FOLLOWER
I20250813 19:58:47.563454 14116 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:58:47.563962 14116 raft_consensus.cc:397] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:58:47.564254 14116 raft_consensus.cc:491] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:58:47.564569 14116 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:47.568871 14116 raft_consensus.cc:513] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:58:47.569535 14116 leader_election.cc:304] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecf90acd0b29462196a47a7b543701c8; no voters:
I20250813 19:58:47.571095 14116 leader_election.cc:290] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 19:58:47.571858 14121 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:47.574240 14121 raft_consensus.cc:695] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 1 LEADER]: Becoming Leader. State: Replica: ecf90acd0b29462196a47a7b543701c8, State: Running, Role: LEADER
I20250813 19:58:47.575030 14116 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 19:58:47.574986 14121 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:58:47.584167 14123 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf90acd0b29462196a47a7b543701c8. Latest consensus state: current_term: 1 leader_uuid: "ecf90acd0b29462196a47a7b543701c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } } }
I20250813 19:58:47.584472 14122 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecf90acd0b29462196a47a7b543701c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } } }
I20250813 19:58:47.584987 14123 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: This master's current role is: LEADER
I20250813 19:58:47.585294 14122 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: This master's current role is: LEADER
I20250813 19:58:47.591816 14129 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 19:58:47.600500 14129 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 19:58:47.618402 14129 catalog_manager.cc:1349] Generated new cluster ID: 0b3991ece4a34d2ea5bc387eb9c2b9d2
I20250813 19:58:47.618662 14129 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 19:58:47.649449 14129 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 19:58:47.650858 14129 catalog_manager.cc:1506] Loading token signing keys...
I20250813 19:58:47.667567 14129 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Generated new TSK 0
I20250813 19:58:47.668463 14129 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 19:58:47.680795 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:0
--local_ip_for_outbound_sockets=127.11.36.193
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:40459
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250813 19:58:48.019655 14140 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:48.020190 14140 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:48.020648 14140 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:48.049283 14140 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:48.050092 14140 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 19:58:48.081743 14140 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:40459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:48.082978 14140 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:48.084509 14140 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:48.096807 14146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:48.097983 14147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:48.103324 14149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:49.497630 14148 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1397 milliseconds
I20250813 19:58:49.497759 14140 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:49.498863 14140 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:49.500993 14140 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:49.502398 14140 hybrid_clock.cc:648] HybridClock initialized: now 1755115129502345 us; error 65 us; skew 500 ppm
I20250813 19:58:49.503360 14140 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:49.510510 14140 webserver.cc:480] Webserver started at http://127.11.36.193:44953/ using document root <none> and password file <none>
I20250813 19:58:49.511370 14140 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:49.511580 14140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:49.512035 14140 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:49.516160 14140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "af1510fe91334bcd8270e3ea9295b2db"
format_stamp: "Formatted at 2025-08-13 19:58:49 on dist-test-slave-l098"
I20250813 19:58:49.517189 14140 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "af1510fe91334bcd8270e3ea9295b2db"
format_stamp: "Formatted at 2025-08-13 19:58:49 on dist-test-slave-l098"
I20250813 19:58:49.523624 14140 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250813 19:58:49.528980 14156 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:49.529975 14140 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250813 19:58:49.530284 14140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "af1510fe91334bcd8270e3ea9295b2db"
format_stamp: "Formatted at 2025-08-13 19:58:49 on dist-test-slave-l098"
I20250813 19:58:49.530606 14140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:49.602533 14140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:49.604024 14140 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:49.604468 14140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:49.606807 14140 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:49.610829 14140 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:49.611023 14140 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:49.611251 14140 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:49.611397 14140 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:49.738598 14140 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:42107
I20250813 19:58:49.738682 14268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:42107 every 8 connection(s)
I20250813 19:58:49.741042 14140 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 19:58:49.743129 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 14140
I20250813 19:58:49.743664 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 19:58:49.750360 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.194:0
--local_ip_for_outbound_sockets=127.11.36.194
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:40459
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:58:49.761027 14269 heartbeater.cc:344] Connected to a master server at 127.11.36.254:40459
I20250813 19:58:49.761426 14269 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:49.762370 14269 heartbeater.cc:507] Master 127.11.36.254:40459 requested a full tablet report, sending...
I20250813 19:58:49.764830 14081 ts_manager.cc:194] Registered new tserver with Master: af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107)
I20250813 19:58:49.767629 14081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:52131
W20250813 19:58:50.030483 14273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:50.030926 14273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:50.031320 14273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:50.060603 14273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:50.062085 14273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.194
I20250813 19:58:50.094570 14273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:40459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.194
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:50.095824 14273 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:50.097230 14273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:50.108966 14279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:58:50.771091 14269 heartbeater.cc:499] Master 127.11.36.254:40459 was elected leader, sending a full tablet report...
W20250813 19:58:50.109417 14280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:51.200469 14282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:51.202793 14281 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1089 milliseconds
I20250813 19:58:51.202883 14273 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:51.203969 14273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:51.205965 14273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:51.207263 14273 hybrid_clock.cc:648] HybridClock initialized: now 1755115131207208 us; error 53 us; skew 500 ppm
I20250813 19:58:51.207959 14273 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:51.213737 14273 webserver.cc:480] Webserver started at http://127.11.36.194:41215/ using document root <none> and password file <none>
I20250813 19:58:51.214538 14273 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:51.214752 14273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:51.215145 14273 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:51.219002 14273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "75a403c4ca9b40a782c8146b272bcd9e"
format_stamp: "Formatted at 2025-08-13 19:58:51 on dist-test-slave-l098"
I20250813 19:58:51.219935 14273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "75a403c4ca9b40a782c8146b272bcd9e"
format_stamp: "Formatted at 2025-08-13 19:58:51 on dist-test-slave-l098"
I20250813 19:58:51.226101 14273 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250813 19:58:51.231276 14289 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:51.232239 14273 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250813 19:58:51.232540 14273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "75a403c4ca9b40a782c8146b272bcd9e"
format_stamp: "Formatted at 2025-08-13 19:58:51 on dist-test-slave-l098"
I20250813 19:58:51.232837 14273 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:51.276252 14273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:51.277628 14273 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:51.278012 14273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:51.280288 14273 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:51.283958 14273 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:51.284180 14273 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250813 19:58:51.284402 14273 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:51.284557 14273 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:51.406116 14273 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.194:43855
I20250813 19:58:51.406205 14401 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.194:43855 every 8 connection(s)
I20250813 19:58:51.408502 14273 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 19:58:51.415539 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 14273
I20250813 19:58:51.415911 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 19:58:51.420961 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.195:0
--local_ip_for_outbound_sockets=127.11.36.195
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:40459
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:58:51.427654 14402 heartbeater.cc:344] Connected to a master server at 127.11.36.254:40459
I20250813 19:58:51.428102 14402 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:51.429054 14402 heartbeater.cc:507] Master 127.11.36.254:40459 requested a full tablet report, sending...
I20250813 19:58:51.431090 14081 ts_manager.cc:194] Registered new tserver with Master: 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855)
I20250813 19:58:51.432883 14081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.194:45029
W20250813 19:58:51.700481 14406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:51.700955 14406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:51.701398 14406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:51.730579 14406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:51.731426 14406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.195
I20250813 19:58:51.763217 14406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:40459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.195
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:51.764590 14406 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:51.766096 14406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:51.777623 14413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:58:52.436548 14402 heartbeater.cc:499] Master 127.11.36.254:40459 was elected leader, sending a full tablet report...
W20250813 19:58:51.779024 14414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:52.952899 14416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:52.957873 14415 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1178 milliseconds
W20250813 19:58:52.958595 14406 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.181s user 0.395s sys 0.780s
W20250813 19:58:52.958811 14406 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.181s user 0.395s sys 0.780s
I20250813 19:58:52.959012 14406 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:58:52.959977 14406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:58:52.962186 14406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:58:52.963584 14406 hybrid_clock.cc:648] HybridClock initialized: now 1755115132963542 us; error 43 us; skew 500 ppm
I20250813 19:58:52.964522 14406 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:58:52.971493 14406 webserver.cc:480] Webserver started at http://127.11.36.195:44991/ using document root <none> and password file <none>
I20250813 19:58:52.972522 14406 fs_manager.cc:362] Metadata directory not provided
I20250813 19:58:52.972723 14406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:58:52.973121 14406 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:58:52.978730 14406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "aa8e1adbb10c4e409605bdc9e95e29b2"
format_stamp: "Formatted at 2025-08-13 19:58:52 on dist-test-slave-l098"
I20250813 19:58:52.979746 14406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "aa8e1adbb10c4e409605bdc9e95e29b2"
format_stamp: "Formatted at 2025-08-13 19:58:52 on dist-test-slave-l098"
I20250813 19:58:52.987008 14406 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250813 19:58:52.992383 14423 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:52.993391 14406 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20250813 19:58:52.993711 14406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "aa8e1adbb10c4e409605bdc9e95e29b2"
format_stamp: "Formatted at 2025-08-13 19:58:52 on dist-test-slave-l098"
I20250813 19:58:52.994033 14406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:58:53.054960 14406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:58:53.056341 14406 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:58:53.056727 14406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:58:53.058995 14406 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:58:53.062680 14406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:58:53.062892 14406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:53.063112 14406 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:58:53.063251 14406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:58:53.189949 14406 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.195:42161
I20250813 19:58:53.190045 14535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.195:42161 every 8 connection(s)
I20250813 19:58:53.192454 14406 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 19:58:53.201421 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 14406
I20250813 19:58:53.201944 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 19:58:53.216516 14536 heartbeater.cc:344] Connected to a master server at 127.11.36.254:40459
I20250813 19:58:53.216931 14536 heartbeater.cc:461] Registering TS with master...
I20250813 19:58:53.217880 14536 heartbeater.cc:507] Master 127.11.36.254:40459 requested a full tablet report, sending...
I20250813 19:58:53.219863 14081 ts_manager.cc:194] Registered new tserver with Master: aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161)
I20250813 19:58:53.221174 14081 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.195:59539
I20250813 19:58:53.221226 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 19:58:53.254321 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 19:58:53.254643 11411 test_util.cc:276] Using random seed: -1777419294
I20250813 19:58:53.291208 14081 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41894:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250813 19:58:53.329243 14337 tablet_service.cc:1468] Processing CreateTablet for tablet 813334b0164e47a4aec2a01ec215d70f (DEFAULT_TABLE table=TestTable [id=f446cc06b4934f14b716bbe9089f8fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:53.330616 14337 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813334b0164e47a4aec2a01ec215d70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:53.348960 14556 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap starting.
I20250813 19:58:53.353940 14556 tablet_bootstrap.cc:654] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:53.355515 14556 log.cc:826] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:53.359911 14556 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: No bootstrap required, opened a new log
I20250813 19:58:53.360323 14556 ts_tablet_manager.cc:1397] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.002s
I20250813 19:58:53.376621 14556 raft_consensus.cc:357] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } }
I20250813 19:58:53.377218 14556 raft_consensus.cc:383] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:53.377472 14556 raft_consensus.cc:738] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a403c4ca9b40a782c8146b272bcd9e, State: Initialized, Role: FOLLOWER
I20250813 19:58:53.378131 14556 consensus_queue.cc:260] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } }
I20250813 19:58:53.378583 14556 raft_consensus.cc:397] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:58:53.378846 14556 raft_consensus.cc:491] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:58:53.379125 14556 raft_consensus.cc:3058] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:53.382812 14556 raft_consensus.cc:513] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } }
I20250813 19:58:53.383486 14556 leader_election.cc:304] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a403c4ca9b40a782c8146b272bcd9e; no voters:
I20250813 19:58:53.385382 14556 leader_election.cc:290] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 19:58:53.385694 14558 raft_consensus.cc:2802] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:53.388152 14558 raft_consensus.cc:695] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 1 LEADER]: Becoming Leader. State: Replica: 75a403c4ca9b40a782c8146b272bcd9e, State: Running, Role: LEADER
I20250813 19:58:53.388942 14558 consensus_queue.cc:237] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } }
I20250813 19:58:53.389333 14556 ts_tablet_manager.cc:1428] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Time spent starting tablet: real 0.029s user 0.018s sys 0.010s
I20250813 19:58:53.400250 14081 catalog_manager.cc:5582] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: term changed from 0 to 1, leader changed from <none> to 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194). New cstate: current_term: 1 leader_uuid: "75a403c4ca9b40a782c8146b272bcd9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } health_report { overall_health: HEALTHY } } }
I20250813 19:58:53.554724 11411 test_util.cc:276] Using random seed: -1777119222
I20250813 19:58:53.575431 14080 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41908:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250813 19:58:53.601413 14204 tablet_service.cc:1468] Processing CreateTablet for tablet 52fe7343bf7f4f99a71f65498175f6a4 (DEFAULT_TABLE table=TestTable1 [id=39e8ec36ae194612ba2e2c93a764959b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:53.602821 14204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52fe7343bf7f4f99a71f65498175f6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:53.620289 14577 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Bootstrap starting.
I20250813 19:58:53.625389 14577 tablet_bootstrap.cc:654] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:53.626999 14577 log.cc:826] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:53.630767 14577 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: No bootstrap required, opened a new log
I20250813 19:58:53.631142 14577 ts_tablet_manager.cc:1397] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20250813 19:58:53.647569 14577 raft_consensus.cc:357] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } }
I20250813 19:58:53.648098 14577 raft_consensus.cc:383] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:53.648304 14577 raft_consensus.cc:738] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1510fe91334bcd8270e3ea9295b2db, State: Initialized, Role: FOLLOWER
I20250813 19:58:53.648902 14577 consensus_queue.cc:260] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } }
I20250813 19:58:53.649344 14577 raft_consensus.cc:397] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:58:53.649590 14577 raft_consensus.cc:491] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:58:53.649863 14577 raft_consensus.cc:3058] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:53.653846 14577 raft_consensus.cc:513] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } }
I20250813 19:58:53.654495 14577 leader_election.cc:304] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af1510fe91334bcd8270e3ea9295b2db; no voters:
I20250813 19:58:53.656183 14577 leader_election.cc:290] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 19:58:53.656524 14579 raft_consensus.cc:2802] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:53.659013 14579 raft_consensus.cc:695] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 1 LEADER]: Becoming Leader. State: Replica: af1510fe91334bcd8270e3ea9295b2db, State: Running, Role: LEADER
I20250813 19:58:53.660166 14577 ts_tablet_manager.cc:1428] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Time spent starting tablet: real 0.029s user 0.020s sys 0.011s
I20250813 19:58:53.660089 14579 consensus_queue.cc:237] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } }
I20250813 19:58:53.671131 14080 catalog_manager.cc:5582] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db reported cstate change: term changed from 0 to 1, leader changed from <none> to af1510fe91334bcd8270e3ea9295b2db (127.11.36.193). New cstate: current_term: 1 leader_uuid: "af1510fe91334bcd8270e3ea9295b2db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } health_report { overall_health: HEALTHY } } }
I20250813 19:58:53.839691 11411 test_util.cc:276] Using random seed: -1776834259
I20250813 19:58:53.858945 14074 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41922:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250813 19:58:53.884984 14471 tablet_service.cc:1468] Processing CreateTablet for tablet 32c153406cdc460d9816ab3f39f5a7f6 (DEFAULT_TABLE table=TestTable2 [id=a33c30ea17f64ff7be501b6f71059f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:58:53.886322 14471 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32c153406cdc460d9816ab3f39f5a7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:58:53.903239 14598 tablet_bootstrap.cc:492] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap starting.
I20250813 19:58:53.908324 14598 tablet_bootstrap.cc:654] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Neither blocks nor log segments found. Creating new log.
I20250813 19:58:53.909850 14598 log.cc:826] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Log is configured to *not* fsync() on all Append() calls
I20250813 19:58:53.913823 14598 tablet_bootstrap.cc:492] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: No bootstrap required, opened a new log
I20250813 19:58:53.914163 14598 ts_tablet_manager.cc:1397] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20250813 19:58:53.930502 14598 raft_consensus.cc:357] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:58:53.931033 14598 raft_consensus.cc:383] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:58:53.931247 14598 raft_consensus.cc:738] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8e1adbb10c4e409605bdc9e95e29b2, State: Initialized, Role: FOLLOWER
I20250813 19:58:53.931844 14598 consensus_queue.cc:260] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:58:53.932338 14598 raft_consensus.cc:397] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:58:53.932586 14598 raft_consensus.cc:491] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:58:53.932857 14598 raft_consensus.cc:3058] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:58:53.936805 14598 raft_consensus.cc:513] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:58:53.937407 14598 leader_election.cc:304] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8e1adbb10c4e409605bdc9e95e29b2; no voters:
I20250813 19:58:53.939013 14598 leader_election.cc:290] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 19:58:53.939533 14600 raft_consensus.cc:2802] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:58:53.941592 14600 raft_consensus.cc:695] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 1 LEADER]: Becoming Leader. State: Replica: aa8e1adbb10c4e409605bdc9e95e29b2, State: Running, Role: LEADER
I20250813 19:58:53.942467 14598 ts_tablet_manager.cc:1428] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent starting tablet: real 0.028s user 0.027s sys 0.002s
I20250813 19:58:53.942709 14536 heartbeater.cc:499] Master 127.11.36.254:40459 was elected leader, sending a full tablet report...
I20250813 19:58:53.942591 14600 consensus_queue.cc:237] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
W20250813 19:58:53.947450 14537 tablet.cc:2378] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:58:53.951671 14074 catalog_manager.cc:5582] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195). New cstate: current_term: 1 leader_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } health_report { overall_health: HEALTHY } } }
I20250813 19:58:54.085314 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 14048
W20250813 19:58:54.415133 14402 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:40459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:40459: connect: Connection refused (error 111)
W20250813 19:58:54.690775 14269 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:40459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:40459: connect: Connection refused (error 111)
W20250813 19:58:54.967123 14536 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:40459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:40459: connect: Connection refused (error 111)
I20250813 19:58:58.570047 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 14140
I20250813 19:58:58.591207 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 14273
I20250813 19:58:58.616014 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 14406
I20250813 19:58:58.641304 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:40459
--webserver_interface=127.11.36.254
--webserver_port=43741
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:40459 with env {}
W20250813 19:58:58.920096 14675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:58:58.920667 14675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:58:58.921103 14675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:58:58.950587 14675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 19:58:58.950934 14675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:58:58.951212 14675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 19:58:58.951436 14675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 19:58:58.983079 14675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:40459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=43741
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:58:58.984336 14675 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:58:58.985896 14675 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:58:58.996814 14681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:58.997267 14682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:58:59.000773 14684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:00.105414 14683 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1104 milliseconds
I20250813 19:59:00.105544 14675 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:00.106786 14675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:00.109191 14675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:00.110499 14675 hybrid_clock.cc:648] HybridClock initialized: now 1755115140110478 us; error 43 us; skew 500 ppm
I20250813 19:59:00.111215 14675 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:00.117714 14675 webserver.cc:480] Webserver started at http://127.11.36.254:43741/ using document root <none> and password file <none>
I20250813 19:59:00.118546 14675 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:00.118731 14675 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:00.125937 14675 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.005s
I20250813 19:59:00.129995 14691 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:00.130882 14675 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250813 19:59:00.131186 14675 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ecf90acd0b29462196a47a7b543701c8"
format_stamp: "Formatted at 2025-08-13 19:58:47 on dist-test-slave-l098"
I20250813 19:59:00.133028 14675 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:00.191237 14675 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:00.192770 14675 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:00.193202 14675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:00.259486 14675 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:40459
I20250813 19:59:00.259565 14742 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:40459 every 8 connection(s)
I20250813 19:59:00.262488 14675 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 19:59:00.266813 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 14675
I20250813 19:59:00.268132 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:42107
--local_ip_for_outbound_sockets=127.11.36.193
--tserver_master_addrs=127.11.36.254:40459
--webserver_port=44953
--webserver_interface=127.11.36.193
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:59:00.272940 14743 sys_catalog.cc:263] Verifying existing consensus state
I20250813 19:59:00.280460 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Bootstrap starting.
I20250813 19:59:00.296522 14743 log.cc:826] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:00.360167 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:00.360994 14743 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Bootstrap complete.
I20250813 19:59:00.381249 14743 raft_consensus.cc:357] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:00.383271 14743 raft_consensus.cc:738] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf90acd0b29462196a47a7b543701c8, State: Initialized, Role: FOLLOWER
I20250813 19:59:00.384012 14743 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:00.384531 14743 raft_consensus.cc:397] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:59:00.384801 14743 raft_consensus.cc:491] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:59:00.385106 14743 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 2 FOLLOWER]: Advancing to term 3
I20250813 19:59:00.390789 14743 raft_consensus.cc:513] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:00.391391 14743 leader_election.cc:304] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecf90acd0b29462196a47a7b543701c8; no voters:
I20250813 19:59:00.393519 14743 leader_election.cc:290] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250813 19:59:00.393915 14747 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 3 FOLLOWER]: Leader election won for term 3
I20250813 19:59:00.397272 14747 raft_consensus.cc:695] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 3 LEADER]: Becoming Leader. State: Replica: ecf90acd0b29462196a47a7b543701c8, State: Running, Role: LEADER
I20250813 19:59:00.398128 14747 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:00.398524 14743 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 19:59:00.408811 14748 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "ecf90acd0b29462196a47a7b543701c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } } }
I20250813 19:59:00.410841 14748 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: This master's current role is: LEADER
I20250813 19:59:00.410197 14749 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf90acd0b29462196a47a7b543701c8. Latest consensus state: current_term: 3 leader_uuid: "ecf90acd0b29462196a47a7b543701c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } } }
I20250813 19:59:00.412664 14749 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: This master's current role is: LEADER
I20250813 19:59:00.415807 14753 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 19:59:00.430099 14753 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=74ef817d031a4838bfac0e056990b56e]
I20250813 19:59:00.431802 14753 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=a33c30ea17f64ff7be501b6f71059f27]
I20250813 19:59:00.433349 14753 catalog_manager.cc:671] Loaded metadata for table TestTable [id=c17f020ca36746b9be6cb22b3979cf18]
I20250813 19:59:00.440853 14753 tablet_loader.cc:96] loaded metadata for tablet 32c153406cdc460d9816ab3f39f5a7f6 (table TestTable2 [id=a33c30ea17f64ff7be501b6f71059f27])
I20250813 19:59:00.442087 14753 tablet_loader.cc:96] loaded metadata for tablet 52fe7343bf7f4f99a71f65498175f6a4 (table TestTable1 [id=74ef817d031a4838bfac0e056990b56e])
I20250813 19:59:00.443269 14753 tablet_loader.cc:96] loaded metadata for tablet 813334b0164e47a4aec2a01ec215d70f (table TestTable [id=c17f020ca36746b9be6cb22b3979cf18])
I20250813 19:59:00.444552 14753 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 19:59:00.449436 14753 catalog_manager.cc:1261] Loaded cluster ID: 0b3991ece4a34d2ea5bc387eb9c2b9d2
I20250813 19:59:00.450479 14753 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 19:59:00.458214 14753 catalog_manager.cc:1506] Loading token signing keys...
I20250813 19:59:00.463141 14753 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Loaded TSK: 0
I20250813 19:59:00.464823 14753 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250813 19:59:00.636914 14745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:00.637430 14745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:00.637919 14745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:00.666985 14745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:00.667789 14745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 19:59:00.701030 14745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:42107
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=44953
--tserver_master_addrs=127.11.36.254:40459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:00.702298 14745 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:00.703893 14745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:00.715860 14770 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:00.717629 14771 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:02.076056 14773 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:02.079035 14772 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1354 milliseconds
W20250813 19:59:02.079162 14745 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.362s user 0.423s sys 0.899s
W20250813 19:59:02.079586 14745 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.362s user 0.423s sys 0.900s
I20250813 19:59:02.080027 14745 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:02.081486 14745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:02.084250 14745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:02.085764 14745 hybrid_clock.cc:648] HybridClock initialized: now 1755115142085665 us; error 111 us; skew 500 ppm
I20250813 19:59:02.087003 14745 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:02.096361 14745 webserver.cc:480] Webserver started at http://127.11.36.193:44953/ using document root <none> and password file <none>
I20250813 19:59:02.097859 14745 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:02.098165 14745 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20250813 19:59:02.108175 14739 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 14676
W20250813 19:59:02.108889 14739 debug-util.cc:398] Leaking SignalData structure 0x7b080006f7a0 after lost signal to thread 14742
I20250813 19:59:02.109639 14745 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.008s sys 0.000s
I20250813 19:59:02.129105 14781 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:02.130720 14745 fs_manager.cc:730] Time spent opening block manager: real 0.018s user 0.005s sys 0.011s
I20250813 19:59:02.131248 14745 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "af1510fe91334bcd8270e3ea9295b2db"
format_stamp: "Formatted at 2025-08-13 19:58:49 on dist-test-slave-l098"
I20250813 19:59:02.134227 14745 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:02.243264 14745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:02.244766 14745 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:02.245167 14745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:02.248134 14745 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:02.254321 14788 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 19:59:02.261536 14745 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 19:59:02.261770 14745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250813 19:59:02.262037 14745 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 19:59:02.266666 14745 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 19:59:02.267095 14745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250813 19:59:02.267235 14788 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Bootstrap starting.
I20250813 19:59:02.341769 14788 log.cc:826] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:02.474166 14788 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:02.475375 14788 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Bootstrap complete.
I20250813 19:59:02.477559 14788 ts_tablet_manager.cc:1397] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Time spent bootstrapping tablet: real 0.211s user 0.153s sys 0.051s
I20250813 19:59:02.503790 14788 raft_consensus.cc:357] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } }
I20250813 19:59:02.507368 14788 raft_consensus.cc:738] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1510fe91334bcd8270e3ea9295b2db, State: Initialized, Role: FOLLOWER
I20250813 19:59:02.508893 14788 consensus_queue.cc:260] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } }
I20250813 19:59:02.509783 14788 raft_consensus.cc:397] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:59:02.510227 14788 raft_consensus.cc:491] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:59:02.510710 14788 raft_consensus.cc:3058] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 1 FOLLOWER]: Advancing to term 2
I20250813 19:59:02.520707 14788 raft_consensus.cc:513] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } }
I20250813 19:59:02.521881 14788 leader_election.cc:304] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af1510fe91334bcd8270e3ea9295b2db; no voters:
I20250813 19:59:02.530319 14788 leader_election.cc:290] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 2 election: Requested vote from peers
I20250813 19:59:02.532279 14822 raft_consensus.cc:2802] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Leader election won for term 2
I20250813 19:59:02.537922 14822 raft_consensus.cc:695] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 LEADER]: Becoming Leader. State: Replica: af1510fe91334bcd8270e3ea9295b2db, State: Running, Role: LEADER
I20250813 19:59:02.539721 14822 consensus_queue.cc:237] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } }
I20250813 19:59:02.543712 14788 ts_tablet_manager.cc:1428] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Time spent starting tablet: real 0.066s user 0.039s sys 0.015s
I20250813 19:59:02.686973 14745 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:42107
I20250813 19:59:02.687171 14900 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:42107 every 8 connection(s)
I20250813 19:59:02.689531 14745 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 19:59:02.698026 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 14745
I20250813 19:59:02.703392 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.194:43855
--local_ip_for_outbound_sockets=127.11.36.194
--tserver_master_addrs=127.11.36.254:40459
--webserver_port=41215
--webserver_interface=127.11.36.194
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:59:02.776547 14901 heartbeater.cc:344] Connected to a master server at 127.11.36.254:40459
I20250813 19:59:02.777037 14901 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:02.778280 14901 heartbeater.cc:507] Master 127.11.36.254:40459 requested a full tablet report, sending...
I20250813 19:59:02.783300 14708 ts_manager.cc:194] Registered new tserver with Master: af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107)
I20250813 19:59:02.787216 14708 catalog_manager.cc:5582] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db reported cstate change: term changed from 0 to 2, leader changed from <none> to af1510fe91334bcd8270e3ea9295b2db (127.11.36.193), VOTER af1510fe91334bcd8270e3ea9295b2db (127.11.36.193) added. New cstate: current_term: 2 leader_uuid: "af1510fe91334bcd8270e3ea9295b2db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } health_report { overall_health: HEALTHY } } }
W20250813 19:59:02.855690 14708 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 52fe7343bf7f4f99a71f65498175f6a4 with cas_config_opid_index -1: no extra replica candidate found for tablet 52fe7343bf7f4f99a71f65498175f6a4 (table TestTable1 [id=74ef817d031a4838bfac0e056990b56e]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250813 19:59:02.858718 14708 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:39877
I20250813 19:59:02.863826 14901 heartbeater.cc:499] Master 127.11.36.254:40459 was elected leader, sending a full tablet report...
W20250813 19:59:03.076375 14905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:03.076872 14905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:03.077308 14905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:03.108265 14905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:03.109149 14905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.194
I20250813 19:59:03.144043 14905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.194:43855
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.36.194
--webserver_port=41215
--tserver_master_addrs=127.11.36.254:40459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.194
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:03.145378 14905 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:03.146906 14905 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:03.161661 14916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:03.171234 14915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:03.173888 14918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:59:03.173643 14905 server_base.cc:1047] running on GCE node
I20250813 19:59:04.318086 14905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:04.329237 14905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:04.330608 14905 hybrid_clock.cc:648] HybridClock initialized: now 1755115144330531 us; error 93 us; skew 500 ppm
I20250813 19:59:04.331382 14905 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:04.337978 14905 webserver.cc:480] Webserver started at http://127.11.36.194:41215/ using document root <none> and password file <none>
I20250813 19:59:04.338860 14905 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:04.339103 14905 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:04.347060 14905 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250813 19:59:04.351472 14925 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:04.352465 14905 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250813 19:59:04.352772 14905 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "75a403c4ca9b40a782c8146b272bcd9e"
format_stamp: "Formatted at 2025-08-13 19:58:51 on dist-test-slave-l098"
I20250813 19:59:04.354601 14905 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:04.424547 14905 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:04.426702 14905 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:04.427263 14905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:04.430642 14905 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:04.438150 14932 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 19:59:04.445152 14905 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 19:59:04.445359 14905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250813 19:59:04.445588 14905 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 19:59:04.449937 14905 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 19:59:04.450114 14905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250813 19:59:04.450428 14932 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap starting.
I20250813 19:59:04.506449 14932 log.cc:826] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:04.589495 14932 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:04.590265 14932 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap complete.
I20250813 19:59:04.591445 14932 ts_tablet_manager.cc:1397] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Time spent bootstrapping tablet: real 0.141s user 0.103s sys 0.032s
I20250813 19:59:04.605280 14932 raft_consensus.cc:357] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } }
I20250813 19:59:04.607268 14932 raft_consensus.cc:738] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a403c4ca9b40a782c8146b272bcd9e, State: Initialized, Role: FOLLOWER
I20250813 19:59:04.608284 14932 consensus_queue.cc:260] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } }
I20250813 19:59:04.608734 14932 raft_consensus.cc:397] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:59:04.608978 14932 raft_consensus.cc:491] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:59:04.609270 14932 raft_consensus.cc:3058] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 1 FOLLOWER]: Advancing to term 2
I20250813 19:59:04.615252 14932 raft_consensus.cc:513] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } }
I20250813 19:59:04.615917 14932 leader_election.cc:304] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a403c4ca9b40a782c8146b272bcd9e; no voters:
I20250813 19:59:04.618427 14932 leader_election.cc:290] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 2 election: Requested vote from peers
I20250813 19:59:04.618808 15036 raft_consensus.cc:2802] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Leader election won for term 2
I20250813 19:59:04.622773 14932 ts_tablet_manager.cc:1428] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Time spent starting tablet: real 0.031s user 0.026s sys 0.007s
I20250813 19:59:04.624439 15036 raft_consensus.cc:695] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 LEADER]: Becoming Leader. State: Replica: 75a403c4ca9b40a782c8146b272bcd9e, State: Running, Role: LEADER
I20250813 19:59:04.625510 14905 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.194:43855
I20250813 19:59:04.625780 15042 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.194:43855 every 8 connection(s)
I20250813 19:59:04.625471 15036 consensus_queue.cc:237] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } }
I20250813 19:59:04.629110 14905 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 19:59:04.633257 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 14905
I20250813 19:59:04.635022 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.195:42161
--local_ip_for_outbound_sockets=127.11.36.195
--tserver_master_addrs=127.11.36.254:40459
--webserver_port=44991
--webserver_interface=127.11.36.195
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:59:04.646950 15044 heartbeater.cc:344] Connected to a master server at 127.11.36.254:40459
I20250813 19:59:04.647313 15044 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:04.648222 15044 heartbeater.cc:507] Master 127.11.36.254:40459 requested a full tablet report, sending...
I20250813 19:59:04.651464 14708 ts_manager.cc:194] Registered new tserver with Master: 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855)
I20250813 19:59:04.652412 14708 catalog_manager.cc:5582] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: term changed from 0 to 2, leader changed from <none> to 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194), VOTER 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194) added. New cstate: current_term: 2 leader_uuid: "75a403c4ca9b40a782c8146b272bcd9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } health_report { overall_health: HEALTHY } } }
I20250813 19:59:04.664568 14708 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.194:34167
I20250813 19:59:04.673415 15044 heartbeater.cc:499] Master 127.11.36.254:40459 was elected leader, sending a full tablet report...
I20250813 19:59:04.683566 14995 consensus_queue.cc:237] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: true } }
I20250813 19:59:04.687220 15036 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER af1510fe91334bcd8270e3ea9295b2db (127.11.36.193) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: true } } }
I20250813 19:59:04.696606 14693 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 813334b0164e47a4aec2a01ec215d70f with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250813 19:59:04.700165 14708 catalog_manager.cc:5582] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: config changed from index -1 to 10, NON_VOTER af1510fe91334bcd8270e3ea9295b2db (127.11.36.193) added. New cstate: current_term: 2 leader_uuid: "75a403c4ca9b40a782c8146b272bcd9e" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250813 19:59:04.715030 14926 consensus_peers.cc:489] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e -> Peer af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107): Couldn't send request to peer af1510fe91334bcd8270e3ea9295b2db. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 813334b0164e47a4aec2a01ec215d70f. This is attempt 1: this message will repeat every 5th retry.
W20250813 19:59:04.943207 15049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:04.943704 15049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:04.944262 15049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:04.974807 15049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:04.975668 15049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.195
I20250813 19:59:05.008597 15049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.195:42161
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.36.195
--webserver_port=44991
--tserver_master_addrs=127.11.36.254:40459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.195
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:05.009956 15049 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:05.011499 15049 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:05.022696 15062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:59:05.171108 15068 ts_tablet_manager.cc:927] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Initiating tablet copy from peer 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855)
I20250813 19:59:05.176826 14856 consensus_queue.cc:237] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: NON_VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: true } }
I20250813 19:59:05.193810 15068 tablet_copy_client.cc:323] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: tablet copy: Beginning tablet copy session from remote peer at address 127.11.36.194:43855
I20250813 19:59:05.229816 15073 raft_consensus.cc:2953] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: NON_VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: true } } }
I20250813 19:59:05.244019 14692 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 52fe7343bf7f4f99a71f65498175f6a4 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250813 19:59:05.243871 15015 tablet_copy_service.cc:140] P 75a403c4ca9b40a782c8146b272bcd9e: Received BeginTabletCopySession request for tablet 813334b0164e47a4aec2a01ec215d70f from peer af1510fe91334bcd8270e3ea9295b2db ({username='slave'} at 127.11.36.193:34563)
I20250813 19:59:05.244608 15015 tablet_copy_service.cc:161] P 75a403c4ca9b40a782c8146b272bcd9e: Beginning new tablet copy session on tablet 813334b0164e47a4aec2a01ec215d70f from peer af1510fe91334bcd8270e3ea9295b2db at {username='slave'} at 127.11.36.193:34563: session id = af1510fe91334bcd8270e3ea9295b2db-813334b0164e47a4aec2a01ec215d70f
W20250813 19:59:05.246477 14783 consensus_peers.cc:489] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db -> Peer 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855): Couldn't send request to peer 75a403c4ca9b40a782c8146b272bcd9e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52fe7343bf7f4f99a71f65498175f6a4. This is attempt 1: this message will repeat every 5th retry.
I20250813 19:59:05.254117 15015 tablet_copy_source_session.cc:215] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Tablet Copy: opened 0 blocks and 1 log segments
I20250813 19:59:05.253275 14708 catalog_manager.cc:5582] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db reported cstate change: config changed from index -1 to 8, NON_VOTER 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194) added. New cstate: current_term: 2 leader_uuid: "af1510fe91334bcd8270e3ea9295b2db" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: NON_VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250813 19:59:05.259783 15068 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813334b0164e47a4aec2a01ec215d70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:59:05.305013 15068 tablet_copy_client.cc:806] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: tablet copy: Starting download of 0 data blocks...
I20250813 19:59:05.306613 15068 tablet_copy_client.cc:670] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: tablet copy: Starting download of 1 WAL segments...
I20250813 19:59:05.315361 15068 tablet_copy_client.cc:538] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250813 19:59:05.339875 15068 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Bootstrap starting.
I20250813 19:59:05.539443 15068 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:05.540764 15068 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Bootstrap complete.
I20250813 19:59:05.541613 15068 ts_tablet_manager.cc:1397] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Time spent bootstrapping tablet: real 0.202s user 0.155s sys 0.021s
I20250813 19:59:05.546129 15068 raft_consensus.cc:357] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: true } }
I20250813 19:59:05.547170 15068 raft_consensus.cc:738] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: af1510fe91334bcd8270e3ea9295b2db, State: Initialized, Role: LEARNER
I20250813 19:59:05.547966 15068 consensus_queue.cc:260] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: NON_VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: true } }
I20250813 19:59:05.551154 15068 ts_tablet_manager.cc:1428] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250813 19:59:05.553764 15015 tablet_copy_service.cc:342] P 75a403c4ca9b40a782c8146b272bcd9e: Request end of tablet copy session af1510fe91334bcd8270e3ea9295b2db-813334b0164e47a4aec2a01ec215d70f received from {username='slave'} at 127.11.36.193:34563
I20250813 19:59:05.554386 15015 tablet_copy_service.cc:434] P 75a403c4ca9b40a782c8146b272bcd9e: ending tablet copy session af1510fe91334bcd8270e3ea9295b2db-813334b0164e47a4aec2a01ec215d70f on tablet 813334b0164e47a4aec2a01ec215d70f with peer af1510fe91334bcd8270e3ea9295b2db
I20250813 19:59:05.575644 14856 raft_consensus.cc:1215] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250813 19:59:05.809361 15081 ts_tablet_manager.cc:927] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Initiating tablet copy from peer af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107)
I20250813 19:59:05.818521 15081 tablet_copy_client.cc:323] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: tablet copy: Beginning tablet copy session from remote peer at address 127.11.36.193:42107
I20250813 19:59:05.821401 14876 tablet_copy_service.cc:140] P af1510fe91334bcd8270e3ea9295b2db: Received BeginTabletCopySession request for tablet 52fe7343bf7f4f99a71f65498175f6a4 from peer 75a403c4ca9b40a782c8146b272bcd9e ({username='slave'} at 127.11.36.194:60641)
I20250813 19:59:05.822206 14876 tablet_copy_service.cc:161] P af1510fe91334bcd8270e3ea9295b2db: Beginning new tablet copy session on tablet 52fe7343bf7f4f99a71f65498175f6a4 from peer 75a403c4ca9b40a782c8146b272bcd9e at {username='slave'} at 127.11.36.194:60641: session id = 75a403c4ca9b40a782c8146b272bcd9e-52fe7343bf7f4f99a71f65498175f6a4
I20250813 19:59:05.832901 14876 tablet_copy_source_session.cc:215] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Tablet Copy: opened 0 blocks and 1 log segments
I20250813 19:59:05.838399 15081 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52fe7343bf7f4f99a71f65498175f6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:59:05.861330 15081 tablet_copy_client.cc:806] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: tablet copy: Starting download of 0 data blocks...
I20250813 19:59:05.862198 15081 tablet_copy_client.cc:670] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: tablet copy: Starting download of 1 WAL segments...
I20250813 19:59:05.870786 15081 tablet_copy_client.cc:538] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250813 19:59:05.882828 15081 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap starting.
I20250813 19:59:06.049082 15081 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:06.050441 15081 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap complete.
I20250813 19:59:06.051354 15081 ts_tablet_manager.cc:1397] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Time spent bootstrapping tablet: real 0.169s user 0.130s sys 0.031s
I20250813 19:59:06.054822 15081 raft_consensus.cc:357] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: NON_VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: true } }
I20250813 19:59:06.055809 15081 raft_consensus.cc:738] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 75a403c4ca9b40a782c8146b272bcd9e, State: Initialized, Role: LEARNER
I20250813 19:59:06.056712 15081 consensus_queue.cc:260] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: NON_VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: true } }
I20250813 19:59:06.063256 15081 ts_tablet_manager.cc:1428] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
I20250813 19:59:06.066146 14876 tablet_copy_service.cc:342] P af1510fe91334bcd8270e3ea9295b2db: Request end of tablet copy session 75a403c4ca9b40a782c8146b272bcd9e-52fe7343bf7f4f99a71f65498175f6a4 received from {username='slave'} at 127.11.36.194:60641
I20250813 19:59:06.066787 14876 tablet_copy_service.cc:434] P af1510fe91334bcd8270e3ea9295b2db: ending tablet copy session 75a403c4ca9b40a782c8146b272bcd9e-52fe7343bf7f4f99a71f65498175f6a4 on tablet 52fe7343bf7f4f99a71f65498175f6a4 with peer 75a403c4ca9b40a782c8146b272bcd9e
I20250813 19:59:06.169368 15079 raft_consensus.cc:1062] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: attempting to promote NON_VOTER af1510fe91334bcd8270e3ea9295b2db to VOTER
I20250813 19:59:06.172705 15079 consensus_queue.cc:237] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } }
I20250813 19:59:06.189111 14856 raft_consensus.cc:1273] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 LEARNER]: Refusing update from remote peer 75a403c4ca9b40a782c8146b272bcd9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250813 19:59:06.191694 15084 consensus_queue.cc:1035] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250813 19:59:06.221412 15079 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, af1510fe91334bcd8270e3ea9295b2db (127.11.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } }
I20250813 19:59:06.225526 14856 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, af1510fe91334bcd8270e3ea9295b2db (127.11.36.193) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } }
I20250813 19:59:06.257644 14708 catalog_manager.cc:5582] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: config changed from index 10 to 11, af1510fe91334bcd8270e3ea9295b2db (127.11.36.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "75a403c4ca9b40a782c8146b272bcd9e" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250813 19:59:06.348493 14995 raft_consensus.cc:1215] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
W20250813 19:59:06.427423 15061 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 15049
W20250813 19:59:06.536489 15061 kernel_stack_watchdog.cc:198] Thread 15049 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250813 19:59:05.022972 15063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:06.537730 15049 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.515s user 0.528s sys 0.869s
W20250813 19:59:06.538421 15049 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.516s user 0.528s sys 0.869s
W20250813 19:59:06.540364 15065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:06.542253 15064 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1514 milliseconds
I20250813 19:59:06.542279 15049 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:06.543443 15049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:06.545498 15049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:06.546813 15049 hybrid_clock.cc:648] HybridClock initialized: now 1755115146546765 us; error 59 us; skew 500 ppm
I20250813 19:59:06.547693 15049 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:06.553406 15049 webserver.cc:480] Webserver started at http://127.11.36.195:44991/ using document root <none> and password file <none>
I20250813 19:59:06.554283 15049 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:06.554507 15049 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:06.561924 15049 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250813 19:59:06.566251 15096 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:06.567238 15049 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250813 19:59:06.567530 15049 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "aa8e1adbb10c4e409605bdc9e95e29b2"
format_stamp: "Formatted at 2025-08-13 19:58:52 on dist-test-slave-l098"
I20250813 19:59:06.569371 15049 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:06.614588 15049 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:06.616036 15049 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:06.616459 15049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:06.618724 15049 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:06.625358 15103 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 19:59:06.632468 15049 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 19:59:06.632790 15049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250813 19:59:06.633106 15049 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 19:59:06.639596 15049 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 19:59:06.639820 15049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.003s sys 0.001s
I20250813 19:59:06.640333 15103 tablet_bootstrap.cc:492] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap starting.
I20250813 19:59:06.714257 15103 log.cc:826] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:06.772878 15090 raft_consensus.cc:1062] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: attempting to promote NON_VOTER 75a403c4ca9b40a782c8146b272bcd9e to VOTER
I20250813 19:59:06.774619 15090 consensus_queue.cc:237] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } }
I20250813 19:59:06.781934 14995 raft_consensus.cc:1273] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 LEARNER]: Refusing update from remote peer af1510fe91334bcd8270e3ea9295b2db: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20250813 19:59:06.783428 15073 consensus_queue.cc:1035] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20250813 19:59:06.792503 15073 raft_consensus.cc:2953] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } }
I20250813 19:59:06.797859 14995 raft_consensus.cc:2953] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } }
I20250813 19:59:06.805696 14707 catalog_manager.cc:5582] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db reported cstate change: config changed from index 8 to 9, 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "af1510fe91334bcd8270e3ea9295b2db" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250813 19:59:06.818095 15103 tablet_bootstrap.cc:492] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:06.819026 15103 tablet_bootstrap.cc:492] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap complete.
I20250813 19:59:06.820644 15103 ts_tablet_manager.cc:1397] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent bootstrapping tablet: real 0.181s user 0.123s sys 0.045s
I20250813 19:59:06.840617 15103 raft_consensus.cc:357] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:59:06.842383 15103 raft_consensus.cc:738] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8e1adbb10c4e409605bdc9e95e29b2, State: Initialized, Role: FOLLOWER
I20250813 19:59:06.843108 15103 consensus_queue.cc:260] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:59:06.843549 15103 raft_consensus.cc:397] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:59:06.843779 15103 raft_consensus.cc:491] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:59:06.844085 15103 raft_consensus.cc:3058] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 1 FOLLOWER]: Advancing to term 2
I20250813 19:59:06.849871 15103 raft_consensus.cc:513] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:59:06.850472 15103 leader_election.cc:304] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8e1adbb10c4e409605bdc9e95e29b2; no voters:
I20250813 19:59:06.852432 15103 leader_election.cc:290] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250813 19:59:06.852694 15196 raft_consensus.cc:2802] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 19:59:06.856138 15103 ts_tablet_manager.cc:1428] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent starting tablet: real 0.035s user 0.027s sys 0.008s
I20250813 19:59:06.856912 15196 raft_consensus.cc:695] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 LEADER]: Becoming Leader. State: Replica: aa8e1adbb10c4e409605bdc9e95e29b2, State: Running, Role: LEADER
I20250813 19:59:06.857919 15196 consensus_queue.cc:237] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:59:06.880426 15049 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.195:42161
I20250813 19:59:06.880990 15220 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.195:42161 every 8 connection(s)
I20250813 19:59:06.882802 15049 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 19:59:06.884356 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 15049
I20250813 19:59:06.907078 15221 heartbeater.cc:344] Connected to a master server at 127.11.36.254:40459
I20250813 19:59:06.907128 14694 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 813334b0164e47a4aec2a01ec215d70f with cas_config_opid_index 10: aborting the task: latest config opid_index 11; task opid_index 10
I20250813 19:59:06.907470 15221 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:06.908501 15221 heartbeater.cc:507] Master 127.11.36.254:40459 requested a full tablet report, sending...
I20250813 19:59:06.911687 14707 ts_manager.cc:194] Registered new tserver with Master: aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161)
I20250813 19:59:06.912629 14707 catalog_manager.cc:5582] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } health_report { overall_health: HEALTHY } } }
I20250813 19:59:06.919144 14707 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.195:48563
I20250813 19:59:06.922286 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 19:59:06.922616 15221 heartbeater.cc:499] Master 127.11.36.254:40459 was elected leader, sending a full tablet report...
I20250813 19:59:06.926689 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250813 19:59:06.930601 11411 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250813 19:59:06.931820 14995 consensus_queue.cc:237] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } }
I20250813 19:59:06.936969 14856 raft_consensus.cc:1273] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Refusing update from remote peer 75a403c4ca9b40a782c8146b272bcd9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250813 19:59:06.940079 15079 consensus_queue.cc:1035] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250813 19:59:06.945431 15227 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } } }
I20250813 19:59:06.947099 14856 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } } }
I20250813 19:59:06.953462 14693 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 813334b0164e47a4aec2a01ec215d70f with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
W20250813 19:59:06.954561 14928 consensus_peers.cc:489] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e -> Peer aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): Couldn't send request to peer aa8e1adbb10c4e409605bdc9e95e29b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 813334b0164e47a4aec2a01ec215d70f. This is attempt 1: this message will repeat every 5th retry.
I20250813 19:59:06.956462 14707 catalog_manager.cc:5582] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: config changed from index 11 to 12, NON_VOTER aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) added. New cstate: current_term: 2 leader_uuid: "75a403c4ca9b40a782c8146b272bcd9e" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250813 19:59:06.976230 14856 consensus_queue.cc:237] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } }
I20250813 19:59:06.980563 14995 raft_consensus.cc:1273] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Refusing update from remote peer af1510fe91334bcd8270e3ea9295b2db: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250813 19:59:06.981528 15073 consensus_queue.cc:1035] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250813 19:59:06.986235 15090 raft_consensus.cc:2953] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } } }
W20250813 19:59:06.988399 14784 consensus_peers.cc:489] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db -> Peer aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): Couldn't send request to peer aa8e1adbb10c4e409605bdc9e95e29b2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 52fe7343bf7f4f99a71f65498175f6a4. This is attempt 1: this message will repeat every 5th retry.
I20250813 19:59:06.987751 14995 raft_consensus.cc:2953] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } } }
I20250813 19:59:06.992547 14692 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 52fe7343bf7f4f99a71f65498175f6a4 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 4)
I20250813 19:59:06.996160 14708 catalog_manager.cc:5582] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db reported cstate change: config changed from index 9 to 10, NON_VOTER aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) added. New cstate: current_term: 2 leader_uuid: "af1510fe91334bcd8270e3ea9295b2db" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250813 19:59:07.357815 15231 ts_tablet_manager.cc:927] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Initiating tablet copy from peer 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855)
I20250813 19:59:07.359594 15231 tablet_copy_client.cc:323] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: tablet copy: Beginning tablet copy session from remote peer at address 127.11.36.194:43855
I20250813 19:59:07.368093 15015 tablet_copy_service.cc:140] P 75a403c4ca9b40a782c8146b272bcd9e: Received BeginTabletCopySession request for tablet 813334b0164e47a4aec2a01ec215d70f from peer aa8e1adbb10c4e409605bdc9e95e29b2 ({username='slave'} at 127.11.36.195:47201)
I20250813 19:59:07.368491 15015 tablet_copy_service.cc:161] P 75a403c4ca9b40a782c8146b272bcd9e: Beginning new tablet copy session on tablet 813334b0164e47a4aec2a01ec215d70f from peer aa8e1adbb10c4e409605bdc9e95e29b2 at {username='slave'} at 127.11.36.195:47201: session id = aa8e1adbb10c4e409605bdc9e95e29b2-813334b0164e47a4aec2a01ec215d70f
I20250813 19:59:07.372558 15015 tablet_copy_source_session.cc:215] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Tablet Copy: opened 0 blocks and 1 log segments
I20250813 19:59:07.374969 15231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813334b0164e47a4aec2a01ec215d70f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:59:07.386415 15231 tablet_copy_client.cc:806] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: tablet copy: Starting download of 0 data blocks...
I20250813 19:59:07.386896 15231 tablet_copy_client.cc:670] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: tablet copy: Starting download of 1 WAL segments...
I20250813 19:59:07.390002 15231 tablet_copy_client.cc:538] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250813 19:59:07.394958 15231 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap starting.
I20250813 19:59:07.435386 15235 ts_tablet_manager.cc:927] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Initiating tablet copy from peer af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107)
I20250813 19:59:07.437565 15235 tablet_copy_client.cc:323] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: tablet copy: Beginning tablet copy session from remote peer at address 127.11.36.193:42107
I20250813 19:59:07.446352 14876 tablet_copy_service.cc:140] P af1510fe91334bcd8270e3ea9295b2db: Received BeginTabletCopySession request for tablet 52fe7343bf7f4f99a71f65498175f6a4 from peer aa8e1adbb10c4e409605bdc9e95e29b2 ({username='slave'} at 127.11.36.195:37649)
I20250813 19:59:07.446775 14876 tablet_copy_service.cc:161] P af1510fe91334bcd8270e3ea9295b2db: Beginning new tablet copy session on tablet 52fe7343bf7f4f99a71f65498175f6a4 from peer aa8e1adbb10c4e409605bdc9e95e29b2 at {username='slave'} at 127.11.36.195:37649: session id = aa8e1adbb10c4e409605bdc9e95e29b2-52fe7343bf7f4f99a71f65498175f6a4
I20250813 19:59:07.451826 14876 tablet_copy_source_session.cc:215] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Tablet Copy: opened 0 blocks and 1 log segments
I20250813 19:59:07.454413 15235 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52fe7343bf7f4f99a71f65498175f6a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:59:07.463533 15235 tablet_copy_client.cc:806] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: tablet copy: Starting download of 0 data blocks...
I20250813 19:59:07.463997 15235 tablet_copy_client.cc:670] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: tablet copy: Starting download of 1 WAL segments...
I20250813 19:59:07.467950 15235 tablet_copy_client.cc:538] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250813 19:59:07.473994 15235 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap starting.
I20250813 19:59:07.489702 15231 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:07.490276 15231 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap complete.
I20250813 19:59:07.490739 15231 ts_tablet_manager.cc:1397] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent bootstrapping tablet: real 0.096s user 0.079s sys 0.016s
I20250813 19:59:07.492285 15231 raft_consensus.cc:357] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } }
I20250813 19:59:07.492724 15231 raft_consensus.cc:738] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: aa8e1adbb10c4e409605bdc9e95e29b2, State: Initialized, Role: LEARNER
I20250813 19:59:07.493219 15231 consensus_queue.cc:260] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } }
I20250813 19:59:07.495988 15231 ts_tablet_manager.cc:1428] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent starting tablet: real 0.005s user 0.006s sys 0.001s
I20250813 19:59:07.497538 15015 tablet_copy_service.cc:342] P 75a403c4ca9b40a782c8146b272bcd9e: Request end of tablet copy session aa8e1adbb10c4e409605bdc9e95e29b2-813334b0164e47a4aec2a01ec215d70f received from {username='slave'} at 127.11.36.195:47201
I20250813 19:59:07.497895 15015 tablet_copy_service.cc:434] P 75a403c4ca9b40a782c8146b272bcd9e: ending tablet copy session aa8e1adbb10c4e409605bdc9e95e29b2-813334b0164e47a4aec2a01ec215d70f on tablet 813334b0164e47a4aec2a01ec215d70f with peer aa8e1adbb10c4e409605bdc9e95e29b2
I20250813 19:59:07.505525 14692 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 52fe7343bf7f4f99a71f65498175f6a4 with cas_config_opid_index 8: aborting the task: latest config opid_index 10; task opid_index 8
I20250813 19:59:07.546003 15235 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:07.546633 15235 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap complete.
I20250813 19:59:07.547089 15235 ts_tablet_manager.cc:1397] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent bootstrapping tablet: real 0.073s user 0.069s sys 0.002s
I20250813 19:59:07.548498 15235 raft_consensus.cc:357] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } }
I20250813 19:59:07.548977 15235 raft_consensus.cc:738] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: aa8e1adbb10c4e409605bdc9e95e29b2, State: Initialized, Role: LEARNER
I20250813 19:59:07.549408 15235 consensus_queue.cc:260] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: NON_VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: true } }
I20250813 19:59:07.550735 15235 ts_tablet_manager.cc:1428] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250813 19:59:07.552008 14876 tablet_copy_service.cc:342] P af1510fe91334bcd8270e3ea9295b2db: Request end of tablet copy session aa8e1adbb10c4e409605bdc9e95e29b2-52fe7343bf7f4f99a71f65498175f6a4 received from {username='slave'} at 127.11.36.195:37649
I20250813 19:59:07.552367 14876 tablet_copy_service.cc:434] P af1510fe91334bcd8270e3ea9295b2db: ending tablet copy session aa8e1adbb10c4e409605bdc9e95e29b2-52fe7343bf7f4f99a71f65498175f6a4 on tablet 52fe7343bf7f4f99a71f65498175f6a4 with peer aa8e1adbb10c4e409605bdc9e95e29b2
I20250813 19:59:07.747869 15167 raft_consensus.cc:1215] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250813 19:59:07.855974 15167 raft_consensus.cc:1215] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250813 19:59:07.934321 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver af1510fe91334bcd8270e3ea9295b2db to finish bootstrapping
I20250813 19:59:07.950078 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 75a403c4ca9b40a782c8146b272bcd9e to finish bootstrapping
I20250813 19:59:07.961517 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver aa8e1adbb10c4e409605bdc9e95e29b2 to finish bootstrapping
I20250813 19:59:08.203039 14975 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 19:59:08.209911 15146 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 19:59:08.210873 15241 raft_consensus.cc:1062] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: attempting to promote NON_VOTER aa8e1adbb10c4e409605bdc9e95e29b2 to VOTER
I20250813 19:59:08.212679 15241 consensus_queue.cc:237] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:08.217852 15166 raft_consensus.cc:1273] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 LEARNER]: Refusing update from remote peer 75a403c4ca9b40a782c8146b272bcd9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250813 19:59:08.220016 15240 consensus_queue.cc:1035] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250813 19:59:08.220248 14856 raft_consensus.cc:1273] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Refusing update from remote peer 75a403c4ca9b40a782c8146b272bcd9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250813 19:59:08.228523 15273 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } } }
I20250813 19:59:08.229730 15240 consensus_queue.cc:1035] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250813 19:59:08.231021 14836 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 19:59:08.230087 15166 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } } }
I20250813 19:59:08.243391 14708 catalog_manager.cc:5582] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: config changed from index 12 to 13, aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "75a403c4ca9b40a782c8146b272bcd9e" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250813 19:59:08.252034 14856 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } } }
I20250813 19:59:08.302047 15243 raft_consensus.cc:1062] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: attempting to promote NON_VOTER aa8e1adbb10c4e409605bdc9e95e29b2 to VOTER
I20250813 19:59:08.303964 15243 consensus_queue.cc:237] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:08.330327 15166 raft_consensus.cc:1273] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 LEARNER]: Refusing update from remote peer af1510fe91334bcd8270e3ea9295b2db: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250813 19:59:08.331697 15073 consensus_queue.cc:1035] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250813 19:59:08.341933 14995 raft_consensus.cc:1273] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Refusing update from remote peer af1510fe91334bcd8270e3ea9295b2db: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250813 19:59:08.343641 15242 consensus_queue.cc:1035] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [LEADER]: Connected to new peer: Peer: permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250813 19:59:08.355896 15073 raft_consensus.cc:2953] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } } }
I20250813 19:59:08.365057 14995 raft_consensus.cc:2953] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } } }
I20250813 19:59:08.368098 15166 raft_consensus.cc:2953] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } } }
I20250813 19:59:08.381651 14707 catalog_manager.cc:5582] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: config changed from index 10 to 11, aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "af1510fe91334bcd8270e3ea9295b2db" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ecf90acd0b29462196a47a7b543701c8 | 127.11.36.254:40459 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.11.36.212:34493 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
75a403c4ca9b40a782c8146b272bcd9e | 127.11.36.194:43855 | HEALTHY | <none> | 1 | 0
aa8e1adbb10c4e409605bdc9e95e29b2 | 127.11.36.195:42161 | HEALTHY | <none> | 1 | 0
af1510fe91334bcd8270e3ea9295b2db | 127.11.36.193:42107 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.11.36.193 | experimental | 127.11.36.193:42107
local_ip_for_outbound_sockets | 127.11.36.194 | experimental | 127.11.36.194:43855
local_ip_for_outbound_sockets | 127.11.36.195 | experimental | 127.11.36.195:42161
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.11.36.193:42107
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.11.36.194:43855
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.11.36.195:42161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.11.36.212:34493 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet 813334b0164e47a4aec2a01ec215d70f of table 'TestTable' is conflicted: 1 replicas' active configs disagree with the leader master's
75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855): RUNNING [LEADER]
af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107): RUNNING
aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): RUNNING [NONVOTER]
All reported replicas are:
A = 75a403c4ca9b40a782c8146b272bcd9e
B = af1510fe91334bcd8270e3ea9295b2db
C = aa8e1adbb10c4e409605bdc9e95e29b2
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C~ | 2 | 12 | Yes
B | A* B C | 2 | 13 | Yes
C | A* B C~ | 2 | 12 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 3 table(s) are not healthy
FAILED
I20250813 19:59:08.523494 11411 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250813 19:59:08.882134 14975 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 19:59:08.901438 14836 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 19:59:08.915513 15146 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ecf90acd0b29462196a47a7b543701c8 | 127.11.36.254:40459 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.11.36.212:34493 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
75a403c4ca9b40a782c8146b272bcd9e | 127.11.36.194:43855 | HEALTHY | <none> | 1 | 0
aa8e1adbb10c4e409605bdc9e95e29b2 | 127.11.36.195:42161 | HEALTHY | <none> | 1 | 0
af1510fe91334bcd8270e3ea9295b2db | 127.11.36.193:42107 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.11.36.193 | experimental | 127.11.36.193:42107
local_ip_for_outbound_sockets | 127.11.36.194 | experimental | 127.11.36.194:43855
local_ip_for_outbound_sockets | 127.11.36.195 | experimental | 127.11.36.195:42161
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.11.36.193:42107
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.11.36.194:43855
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.11.36.195:42161
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.11.36.212:34493 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250813 19:59:09.113992 11411 log_verifier.cc:126] Checking tablet 32c153406cdc460d9816ab3f39f5a7f6
I20250813 19:59:09.142858 11411 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 32c153406cdc460d9816ab3f39f5a7f6
I20250813 19:59:09.143112 11411 log_verifier.cc:126] Checking tablet 52fe7343bf7f4f99a71f65498175f6a4
I20250813 19:59:09.219910 11411 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 52fe7343bf7f4f99a71f65498175f6a4
I20250813 19:59:09.220199 11411 log_verifier.cc:126] Checking tablet 813334b0164e47a4aec2a01ec215d70f
I20250813 19:59:09.298516 11411 log_verifier.cc:177] Verified matching terms for 13 ops in tablet 813334b0164e47a4aec2a01ec215d70f
I20250813 19:59:09.298933 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 14675
I20250813 19:59:09.324236 11411 minidump.cc:252] Setting minidump size limit to 20M
I20250813 19:59:09.325392 11411 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:09.326346 11411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:09.335525 15315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:09.407879 15221 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:40459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:40459: connect: Connection refused (error 111)
W20250813 19:59:09.412729 14901 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:40459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:40459: connect: Connection refused (error 111)
W20250813 19:59:09.335777 15316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:09.337366 15318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:59:09.448524 11411 server_base.cc:1047] running on GCE node
I20250813 19:59:09.449718 11411 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250813 19:59:09.449951 11411 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250813 19:59:09.450119 11411 hybrid_clock.cc:648] HybridClock initialized: now 1755115149450096 us; error 0 us; skew 500 ppm
I20250813 19:59:09.450721 11411 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:09.454167 11411 webserver.cc:480] Webserver started at http://0.0.0.0:33079/ using document root <none> and password file <none>
I20250813 19:59:09.454892 11411 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:09.455070 11411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:09.459892 11411 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20250813 19:59:09.463107 15326 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:09.463959 11411 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20250813 19:59:09.464278 11411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ecf90acd0b29462196a47a7b543701c8"
format_stamp: "Formatted at 2025-08-13 19:58:47 on dist-test-slave-l098"
I20250813 19:59:09.465745 11411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:09.481928 11411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:09.483304 11411 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:09.483736 11411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:09.491854 11411 sys_catalog.cc:263] Verifying existing consensus state
W20250813 19:59:09.495160 11411 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.11.36.254:40459 exists but no master address supplied!
I20250813 19:59:09.496980 11411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Bootstrap starting.
I20250813 19:59:09.534374 11411 log.cc:826] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:09.589390 11411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:09.590101 11411 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Bootstrap complete.
I20250813 19:59:09.601989 11411 raft_consensus.cc:357] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:09.602532 11411 raft_consensus.cc:738] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf90acd0b29462196a47a7b543701c8, State: Initialized, Role: FOLLOWER
I20250813 19:59:09.603260 11411 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:09.603725 11411 raft_consensus.cc:397] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:59:09.603981 11411 raft_consensus.cc:491] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:59:09.604327 11411 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 3 FOLLOWER]: Advancing to term 4
I20250813 19:59:09.609194 11411 raft_consensus.cc:513] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:09.609853 11411 leader_election.cc:304] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecf90acd0b29462196a47a7b543701c8; no voters:
I20250813 19:59:09.610882 11411 leader_election.cc:290] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250813 19:59:09.611179 15333 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 4 FOLLOWER]: Leader election won for term 4
I20250813 19:59:09.612375 15333 raft_consensus.cc:695] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 4 LEADER]: Becoming Leader. State: Replica: ecf90acd0b29462196a47a7b543701c8, State: Running, Role: LEADER
I20250813 19:59:09.613059 15333 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:09.620635 15335 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf90acd0b29462196a47a7b543701c8. Latest consensus state: current_term: 4 leader_uuid: "ecf90acd0b29462196a47a7b543701c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } } }
I20250813 19:59:09.621179 15335 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: This master's current role is: LEADER
I20250813 19:59:09.621601 15334 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "ecf90acd0b29462196a47a7b543701c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } } }
I20250813 19:59:09.621992 15334 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: This master's current role is: LEADER
I20250813 19:59:09.655973 11411 tablet_replica.cc:331] stopping tablet replica
I20250813 19:59:09.656652 11411 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 4 LEADER]: Raft consensus shutting down.
I20250813 19:59:09.657146 11411 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250813 19:59:09.659911 11411 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250813 19:59:09.660555 11411 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250813 19:59:09.691728 11411 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250813 19:59:09.793785 15044 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:40459 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:40459: connect: Connection refused (error 111)
I20250813 19:59:14.551075 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 14745
I20250813 19:59:14.577149 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 14905
I20250813 19:59:14.610215 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 15049
I20250813 19:59:14.639611 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:40459
--webserver_interface=127.11.36.254
--webserver_port=43741
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:40459 with env {}
W20250813 19:59:14.921051 15406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:14.921645 15406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:14.922071 15406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:14.951844 15406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 19:59:14.952215 15406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:14.952459 15406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 19:59:14.952689 15406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 19:59:14.986441 15406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:40459
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:40459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=43741
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:14.987813 15406 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:14.989499 15406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:14.999439 15412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:15.001031 15413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:16.114009 15415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:16.115618 15414 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1111 milliseconds
I20250813 19:59:16.115749 15406 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:16.116931 15406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:16.119799 15406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:16.121282 15406 hybrid_clock.cc:648] HybridClock initialized: now 1755115156121256 us; error 31 us; skew 500 ppm
I20250813 19:59:16.121968 15406 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:16.129206 15406 webserver.cc:480] Webserver started at http://127.11.36.254:43741/ using document root <none> and password file <none>
I20250813 19:59:16.129997 15406 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:16.130178 15406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:16.137550 15406 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250813 19:59:16.141667 15422 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:16.142563 15406 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250813 19:59:16.142839 15406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ecf90acd0b29462196a47a7b543701c8"
format_stamp: "Formatted at 2025-08-13 19:58:47 on dist-test-slave-l098"
I20250813 19:59:16.144840 15406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:16.197154 15406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:16.198509 15406 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:16.198896 15406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:16.264353 15406 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:40459
I20250813 19:59:16.264406 15473 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:40459 every 8 connection(s)
I20250813 19:59:16.266988 15406 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 19:59:16.276006 15474 sys_catalog.cc:263] Verifying existing consensus state
I20250813 19:59:16.277390 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 15406
I20250813 19:59:16.278671 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:42107
--local_ip_for_outbound_sockets=127.11.36.193
--tserver_master_addrs=127.11.36.254:40459
--webserver_port=44953
--webserver_interface=127.11.36.193
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:59:16.280346 15474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Bootstrap starting.
I20250813 19:59:16.289426 15474 log.cc:826] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:16.359508 15474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:16.360262 15474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Bootstrap complete.
I20250813 19:59:16.377399 15474 raft_consensus.cc:357] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:16.379333 15474 raft_consensus.cc:738] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecf90acd0b29462196a47a7b543701c8, State: Initialized, Role: FOLLOWER
I20250813 19:59:16.380055 15474 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:16.380512 15474 raft_consensus.cc:397] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:59:16.380741 15474 raft_consensus.cc:491] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:59:16.381002 15474 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 5 FOLLOWER]: Advancing to term 6
I20250813 19:59:16.385528 15474 raft_consensus.cc:513] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:16.386070 15474 leader_election.cc:304] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecf90acd0b29462196a47a7b543701c8; no voters:
I20250813 19:59:16.388069 15474 leader_election.cc:290] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250813 19:59:16.388446 15478 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 6 FOLLOWER]: Leader election won for term 6
I20250813 19:59:16.391271 15478 raft_consensus.cc:695] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [term 6 LEADER]: Becoming Leader. State: Replica: ecf90acd0b29462196a47a7b543701c8, State: Running, Role: LEADER
I20250813 19:59:16.391997 15478 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } }
I20250813 19:59:16.392624 15474 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 19:59:16.401436 15479 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "ecf90acd0b29462196a47a7b543701c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } } }
I20250813 19:59:16.402194 15479 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: This master's current role is: LEADER
I20250813 19:59:16.403461 15480 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecf90acd0b29462196a47a7b543701c8. Latest consensus state: current_term: 6 leader_uuid: "ecf90acd0b29462196a47a7b543701c8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecf90acd0b29462196a47a7b543701c8" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 40459 } } }
I20250813 19:59:16.404201 15480 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8 [sys.catalog]: This master's current role is: LEADER
I20250813 19:59:16.429206 15494 catalog_manager.cc:1261] Loaded cluster ID: 0b3991ece4a34d2ea5bc387eb9c2b9d2
I20250813 19:59:16.429409 15494 catalog_manager.cc:1554] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: loading cluster ID for follower catalog manager: success
I20250813 19:59:16.434494 15494 catalog_manager.cc:1576] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: acquiring CA information for follower catalog manager: success
I20250813 19:59:16.438630 15494 catalog_manager.cc:1604] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250813 19:59:16.439122 15485 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 19:59:16.444875 15485 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=74ef817d031a4838bfac0e056990b56e]
I20250813 19:59:16.446035 15485 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=a33c30ea17f64ff7be501b6f71059f27]
I20250813 19:59:16.447440 15485 catalog_manager.cc:671] Loaded metadata for table TestTable [id=cfcb0a4a990942dcb80355dd990df81f]
I20250813 19:59:16.452186 15485 tablet_loader.cc:96] loaded metadata for tablet 32c153406cdc460d9816ab3f39f5a7f6 (table TestTable2 [id=a33c30ea17f64ff7be501b6f71059f27])
I20250813 19:59:16.453053 15485 tablet_loader.cc:96] loaded metadata for tablet 52fe7343bf7f4f99a71f65498175f6a4 (table TestTable1 [id=74ef817d031a4838bfac0e056990b56e])
I20250813 19:59:16.454073 15485 tablet_loader.cc:96] loaded metadata for tablet 813334b0164e47a4aec2a01ec215d70f (table TestTable [id=cfcb0a4a990942dcb80355dd990df81f])
I20250813 19:59:16.454823 15485 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 19:59:16.457850 15485 catalog_manager.cc:1261] Loaded cluster ID: 0b3991ece4a34d2ea5bc387eb9c2b9d2
I20250813 19:59:16.458048 15485 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 19:59:16.461535 15485 catalog_manager.cc:1506] Loading token signing keys...
I20250813 19:59:16.464644 15485 catalog_manager.cc:5966] T 00000000000000000000000000000000 P ecf90acd0b29462196a47a7b543701c8: Loaded TSK: 0
I20250813 19:59:16.465521 15485 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250813 19:59:16.603878 15476 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:16.604437 15476 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:16.604897 15476 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:16.634253 15476 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:16.635133 15476 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 19:59:16.668648 15476 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:42107
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=44953
--tserver_master_addrs=127.11.36.254:40459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:16.670023 15476 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:16.671525 15476 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:16.683207 15501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:18.080936 15504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:16.684422 15502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:18.083642 15476 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.400s user 0.462s sys 0.849s
W20250813 19:59:18.084019 15476 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.400s user 0.462s sys 0.850s
W20250813 19:59:18.088579 15503 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1396 milliseconds
I20250813 19:59:18.088608 15476 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:18.090168 15476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:18.092837 15476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:18.094391 15476 hybrid_clock.cc:648] HybridClock initialized: now 1755115158094351 us; error 51 us; skew 500 ppm
I20250813 19:59:18.095618 15476 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:18.104449 15476 webserver.cc:480] Webserver started at http://127.11.36.193:44953/ using document root <none> and password file <none>
I20250813 19:59:18.105713 15476 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:18.105985 15476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:18.116891 15476 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.006s
I20250813 19:59:18.123076 15511 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:18.124379 15476 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250813 19:59:18.124758 15476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "af1510fe91334bcd8270e3ea9295b2db"
format_stamp: "Formatted at 2025-08-13 19:58:49 on dist-test-slave-l098"
I20250813 19:59:18.127414 15476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:18.224227 15476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:18.226190 15476 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:18.226730 15476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:18.229908 15476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:18.237394 15518 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250813 19:59:18.250298 15476 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250813 19:59:18.250591 15476 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.002s sys 0.000s
I20250813 19:59:18.250891 15476 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250813 19:59:18.258175 15518 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Bootstrap starting.
I20250813 19:59:18.262810 15476 ts_tablet_manager.cc:610] Registered 2 tablets
I20250813 19:59:18.263125 15476 ts_tablet_manager.cc:589] Time spent register tablets: real 0.012s user 0.013s sys 0.000s
I20250813 19:59:18.313297 15518 log.cc:826] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:18.425575 15518 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:18.426632 15518 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Bootstrap complete.
I20250813 19:59:18.428740 15518 ts_tablet_manager.cc:1397] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Time spent bootstrapping tablet: real 0.171s user 0.135s sys 0.032s
I20250813 19:59:18.451292 15476 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:42107
I20250813 19:59:18.451407 15625 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:42107 every 8 connection(s)
I20250813 19:59:18.452415 15518 raft_consensus.cc:357] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:18.454455 15518 raft_consensus.cc:738] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1510fe91334bcd8270e3ea9295b2db, State: Initialized, Role: FOLLOWER
I20250813 19:59:18.454881 15476 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 19:59:18.455235 15518 consensus_queue.cc:260] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:18.459688 15518 ts_tablet_manager.cc:1428] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db: Time spent starting tablet: real 0.031s user 0.023s sys 0.004s
I20250813 19:59:18.460445 15518 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Bootstrap starting.
I20250813 19:59:18.463277 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 15476
I20250813 19:59:18.464936 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.194:43855
--local_ip_for_outbound_sockets=127.11.36.194
--tserver_master_addrs=127.11.36.254:40459
--webserver_port=41215
--webserver_interface=127.11.36.194
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:59:18.511358 15627 heartbeater.cc:344] Connected to a master server at 127.11.36.254:40459
I20250813 19:59:18.511837 15627 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:18.513094 15627 heartbeater.cc:507] Master 127.11.36.254:40459 requested a full tablet report, sending...
I20250813 19:59:18.518314 15439 ts_manager.cc:194] Registered new tserver with Master: af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107)
I20250813 19:59:18.524188 15439 catalog_manager.cc:5582] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194) added, VOTER aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) added, VOTER af1510fe91334bcd8270e3ea9295b2db (127.11.36.193) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } } }
I20250813 19:59:18.597066 15439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:59781
I20250813 19:59:18.601473 15627 heartbeater.cc:499] Master 127.11.36.254:40459 was elected leader, sending a full tablet report...
I20250813 19:59:18.647446 15518 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:18.648387 15518 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Bootstrap complete.
I20250813 19:59:18.650058 15518 ts_tablet_manager.cc:1397] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Time spent bootstrapping tablet: real 0.190s user 0.156s sys 0.028s
I20250813 19:59:18.652316 15518 raft_consensus.cc:357] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:18.652958 15518 raft_consensus.cc:738] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1510fe91334bcd8270e3ea9295b2db, State: Initialized, Role: FOLLOWER
I20250813 19:59:18.653575 15518 consensus_queue.cc:260] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:18.655490 15518 ts_tablet_manager.cc:1428] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20250813 19:59:18.825932 15631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:18.826427 15631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:18.826870 15631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:18.855815 15631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:18.856622 15631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.194
I20250813 19:59:18.889056 15631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.194:43855
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.36.194
--webserver_port=41215
--tserver_master_addrs=127.11.36.254:40459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.194
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:18.890283 15631 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:18.891794 15631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:18.903378 15642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:18.904840 15643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:20.060396 15645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:20.064018 15644 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1158 milliseconds
I20250813 19:59:20.064136 15631 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:20.065227 15631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:20.067145 15631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:20.068487 15631 hybrid_clock.cc:648] HybridClock initialized: now 1755115160068444 us; error 43 us; skew 500 ppm
I20250813 19:59:20.069268 15631 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:20.074721 15631 webserver.cc:480] Webserver started at http://127.11.36.194:41215/ using document root <none> and password file <none>
I20250813 19:59:20.075541 15631 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:20.075737 15631 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:20.082592 15631 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20250813 19:59:20.086714 15652 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:20.087630 15631 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250813 19:59:20.087932 15631 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "75a403c4ca9b40a782c8146b272bcd9e"
format_stamp: "Formatted at 2025-08-13 19:58:51 on dist-test-slave-l098"
I20250813 19:59:20.089715 15631 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:20.138793 15631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:20.140209 15631 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:20.140592 15631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:20.142915 15631 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:20.148006 15659 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250813 19:59:20.158221 15631 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250813 19:59:20.158444 15631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.002s sys 0.000s
I20250813 19:59:20.158667 15631 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250813 19:59:20.163308 15659 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap starting.
I20250813 19:59:20.165627 15631 ts_tablet_manager.cc:610] Registered 2 tablets
I20250813 19:59:20.165802 15631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.006s sys 0.000s
I20250813 19:59:20.198257 15673 raft_consensus.cc:491] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 19:59:20.199019 15673 raft_consensus.cc:513] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:20.203650 15673 leader_election.cc:290] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855), aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161)
I20250813 19:59:20.205183 15673 raft_consensus.cc:491] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 19:59:20.205744 15673 raft_consensus.cc:513] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:20.207654 15673 leader_election.cc:290] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855), aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161)
W20250813 19:59:20.207805 15514 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.36.195:42161: connect: Connection refused (error 111)
W20250813 19:59:20.213276 15514 leader_election.cc:336] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): Network error: Client connection negotiation failed: client connection to 127.11.36.195:42161: connect: Connection refused (error 111)
W20250813 19:59:20.214887 15513 leader_election.cc:336] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855): Network error: Client connection negotiation failed: client connection to 127.11.36.194:43855: connect: Connection refused (error 111)
I20250813 19:59:20.215389 15513 leader_election.cc:304] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af1510fe91334bcd8270e3ea9295b2db; no voters: 75a403c4ca9b40a782c8146b272bcd9e, aa8e1adbb10c4e409605bdc9e95e29b2
I20250813 19:59:20.216342 15673 raft_consensus.cc:2747] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250813 19:59:20.216449 15513 leader_election.cc:336] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855): Network error: Client connection negotiation failed: client connection to 127.11.36.194:43855: connect: Connection refused (error 111)
W20250813 19:59:20.218564 15514 leader_election.cc:336] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): Network error: Client connection negotiation failed: client connection to 127.11.36.195:42161: connect: Connection refused (error 111)
I20250813 19:59:20.218976 15514 leader_election.cc:304] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af1510fe91334bcd8270e3ea9295b2db; no voters: 75a403c4ca9b40a782c8146b272bcd9e, aa8e1adbb10c4e409605bdc9e95e29b2
I20250813 19:59:20.219679 15673 raft_consensus.cc:2747] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250813 19:59:20.221540 15659 log.cc:826] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:20.319808 15659 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:20.320600 15659 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap complete.
I20250813 19:59:20.322319 15659 ts_tablet_manager.cc:1397] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Time spent bootstrapping tablet: real 0.159s user 0.113s sys 0.040s
I20250813 19:59:20.338127 15659 raft_consensus.cc:357] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:20.340416 15659 raft_consensus.cc:738] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a403c4ca9b40a782c8146b272bcd9e, State: Initialized, Role: FOLLOWER
I20250813 19:59:20.341224 15659 consensus_queue.cc:260] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:20.345367 15659 ts_tablet_manager.cc:1428] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e: Time spent starting tablet: real 0.023s user 0.020s sys 0.004s
I20250813 19:59:20.345917 15659 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap starting.
I20250813 19:59:20.346357 15631 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.194:43855
I20250813 19:59:20.346872 15771 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.194:43855 every 8 connection(s)
I20250813 19:59:20.349114 15631 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 19:59:20.354470 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 15631
I20250813 19:59:20.356627 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.195:42161
--local_ip_for_outbound_sockets=127.11.36.195
--tserver_master_addrs=127.11.36.254:40459
--webserver_port=44991
--webserver_interface=127.11.36.195
--builtin_ntp_servers=127.11.36.212:34493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250813 19:59:20.385528 15772 heartbeater.cc:344] Connected to a master server at 127.11.36.254:40459
I20250813 19:59:20.385993 15772 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:20.386898 15772 heartbeater.cc:507] Master 127.11.36.254:40459 requested a full tablet report, sending...
I20250813 19:59:20.390244 15438 ts_manager.cc:194] Registered new tserver with Master: 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194:43855)
I20250813 19:59:20.393229 15438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.194:57245
I20250813 19:59:20.507206 15659 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:20.508164 15659 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Bootstrap complete.
I20250813 19:59:20.509807 15659 ts_tablet_manager.cc:1397] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Time spent bootstrapping tablet: real 0.164s user 0.144s sys 0.016s
I20250813 19:59:20.511924 15659 raft_consensus.cc:357] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:20.512583 15659 raft_consensus.cc:738] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a403c4ca9b40a782c8146b272bcd9e, State: Initialized, Role: FOLLOWER
I20250813 19:59:20.513108 15659 consensus_queue.cc:260] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:20.514021 15772 heartbeater.cc:499] Master 127.11.36.254:40459 was elected leader, sending a full tablet report...
I20250813 19:59:20.514564 15659 ts_tablet_manager.cc:1428] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20250813 19:59:20.683845 15776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:20.684345 15776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:20.684775 15776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:20.712527 15776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:20.713260 15776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.195
I20250813 19:59:20.744699 15776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:34493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.195:42161
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.36.195
--webserver_port=44991
--tserver_master_addrs=127.11.36.254:40459
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.195
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:20.745862 15776 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:20.747288 15776 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:20.757921 15783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:59:21.660876 15789 raft_consensus.cc:491] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 19:59:21.662009 15789 raft_consensus.cc:513] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:21.676357 15789 leader_election.cc:290] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107), aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161)
W20250813 19:59:21.690433 15655 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.36.195:42161: connect: Connection refused (error 111)
I20250813 19:59:21.704666 15581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813334b0164e47a4aec2a01ec215d70f" candidate_uuid: "75a403c4ca9b40a782c8146b272bcd9e" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "af1510fe91334bcd8270e3ea9295b2db" is_pre_election: true
I20250813 19:59:21.705582 15581 raft_consensus.cc:2466] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75a403c4ca9b40a782c8146b272bcd9e in term 2.
I20250813 19:59:21.707847 15653 leader_election.cc:304] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75a403c4ca9b40a782c8146b272bcd9e, af1510fe91334bcd8270e3ea9295b2db; no voters:
I20250813 19:59:21.708979 15789 raft_consensus.cc:2802] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250813 19:59:21.709491 15789 raft_consensus.cc:491] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 19:59:21.709918 15789 raft_consensus.cc:3058] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Advancing to term 3
W20250813 19:59:21.710492 15655 leader_election.cc:336] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): Network error: Client connection negotiation failed: client connection to 127.11.36.195:42161: connect: Connection refused (error 111)
I20250813 19:59:21.724839 15789 raft_consensus.cc:513] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:21.728654 15581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813334b0164e47a4aec2a01ec215d70f" candidate_uuid: "75a403c4ca9b40a782c8146b272bcd9e" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "af1510fe91334bcd8270e3ea9295b2db"
I20250813 19:59:21.729404 15581 raft_consensus.cc:3058] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Advancing to term 3
I20250813 19:59:21.742551 15581 raft_consensus.cc:2466] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75a403c4ca9b40a782c8146b272bcd9e in term 3.
W20250813 19:59:21.749578 15655 leader_election.cc:336] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): Network error: Client connection negotiation failed: client connection to 127.11.36.195:42161: connect: Connection refused (error 111)
I20250813 19:59:21.750257 15789 leader_election.cc:290] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 election: Requested vote from peers af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107), aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161)
I20250813 19:59:21.751485 15653 leader_election.cc:304] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75a403c4ca9b40a782c8146b272bcd9e, af1510fe91334bcd8270e3ea9295b2db; no voters: aa8e1adbb10c4e409605bdc9e95e29b2
I20250813 19:59:21.752585 15789 raft_consensus.cc:2802] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 3 FOLLOWER]: Leader election won for term 3
I20250813 19:59:21.768505 15789 raft_consensus.cc:695] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 3 LEADER]: Becoming Leader. State: Replica: 75a403c4ca9b40a782c8146b272bcd9e, State: Running, Role: LEADER
I20250813 19:59:21.769548 15789 consensus_queue.cc:237] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:21.804200 15438 catalog_manager.cc:5582] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: term changed from 2 to 3, leader changed from <none> to 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194). New cstate: current_term: 3 leader_uuid: "75a403c4ca9b40a782c8146b272bcd9e" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250813 19:59:21.889982 15789 raft_consensus.cc:491] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 19:59:21.890437 15789 raft_consensus.cc:513] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:21.891978 15789 leader_election.cc:290] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107), aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161)
I20250813 19:59:21.895265 15581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52fe7343bf7f4f99a71f65498175f6a4" candidate_uuid: "75a403c4ca9b40a782c8146b272bcd9e" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "af1510fe91334bcd8270e3ea9295b2db" is_pre_election: true
I20250813 19:59:21.895743 15581 raft_consensus.cc:2466] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 75a403c4ca9b40a782c8146b272bcd9e in term 2.
I20250813 19:59:21.896689 15653 leader_election.cc:304] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 75a403c4ca9b40a782c8146b272bcd9e, af1510fe91334bcd8270e3ea9295b2db; no voters:
I20250813 19:59:21.898140 15789 raft_consensus.cc:2802] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250813 19:59:21.898458 15789 raft_consensus.cc:491] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 19:59:21.898777 15789 raft_consensus.cc:3058] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 2 FOLLOWER]: Advancing to term 3
I20250813 19:59:21.902479 15789 raft_consensus.cc:513] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:21.903721 15789 leader_election.cc:290] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 election: Requested vote from peers af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107), aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161)
W20250813 19:59:21.904316 15655 leader_election.cc:336] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): Network error: Client connection negotiation failed: client connection to 127.11.36.195:42161: connect: Connection refused (error 111)
I20250813 19:59:21.904531 15581 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52fe7343bf7f4f99a71f65498175f6a4" candidate_uuid: "75a403c4ca9b40a782c8146b272bcd9e" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "af1510fe91334bcd8270e3ea9295b2db"
I20250813 19:59:21.904973 15581 raft_consensus.cc:3058] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 2 FOLLOWER]: Advancing to term 3
W20250813 19:59:21.912565 15655 leader_election.cc:336] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): Network error: Client connection negotiation failed: client connection to 127.11.36.195:42161: connect: Connection refused (error 111)
I20250813 19:59:21.914299 15581 raft_consensus.cc:2466] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 75a403c4ca9b40a782c8146b272bcd9e in term 3.
I20250813 19:59:21.915277 15653 leader_election.cc:304] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 75a403c4ca9b40a782c8146b272bcd9e, af1510fe91334bcd8270e3ea9295b2db; no voters: aa8e1adbb10c4e409605bdc9e95e29b2
I20250813 19:59:21.916195 15789 raft_consensus.cc:2802] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 3 FOLLOWER]: Leader election won for term 3
I20250813 19:59:21.916776 15789 raft_consensus.cc:695] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [term 3 LEADER]: Becoming Leader. State: Replica: 75a403c4ca9b40a782c8146b272bcd9e, State: Running, Role: LEADER
I20250813 19:59:21.917568 15789 consensus_queue.cc:237] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:21.932245 15438 catalog_manager.cc:5582] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: term changed from 2 to 3, leader changed from af1510fe91334bcd8270e3ea9295b2db (127.11.36.193) to 75a403c4ca9b40a782c8146b272bcd9e (127.11.36.194). New cstate: current_term: 3 leader_uuid: "75a403c4ca9b40a782c8146b272bcd9e" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250813 19:59:20.758419 15784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:22.015897 15785 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1255 milliseconds
W20250813 19:59:22.017220 15786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:22.019264 15776 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.261s user 0.451s sys 0.793s
W20250813 19:59:22.019501 15776 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.261s user 0.451s sys 0.793s
I20250813 19:59:22.019677 15776 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:22.020615 15776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:22.022666 15776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:22.023985 15776 hybrid_clock.cc:648] HybridClock initialized: now 1755115162023952 us; error 51 us; skew 500 ppm
I20250813 19:59:22.024739 15776 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:22.031306 15776 webserver.cc:480] Webserver started at http://127.11.36.195:44991/ using document root <none> and password file <none>
I20250813 19:59:22.032320 15776 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:22.032532 15776 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:22.040230 15776 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250813 19:59:22.044814 15806 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:22.045959 15776 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250813 19:59:22.046273 15776 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "aa8e1adbb10c4e409605bdc9e95e29b2"
format_stamp: "Formatted at 2025-08-13 19:58:52 on dist-test-slave-l098"
I20250813 19:59:22.048137 15776 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:22.114741 15776 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:22.116168 15776 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:22.116564 15776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:22.119104 15776 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:22.124502 15813 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250813 19:59:22.139400 15776 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250813 19:59:22.139642 15776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.002s
I20250813 19:59:22.139874 15776 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250813 19:59:22.144796 15813 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap starting.
I20250813 19:59:22.150070 15776 ts_tablet_manager.cc:610] Registered 3 tablets
I20250813 19:59:22.150290 15776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.006s sys 0.003s
I20250813 19:59:22.202986 15813 log.cc:826] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:22.279605 15813 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:22.280665 15813 tablet_bootstrap.cc:492] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap complete.
I20250813 19:59:22.282428 15813 ts_tablet_manager.cc:1397] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent bootstrapping tablet: real 0.138s user 0.100s sys 0.033s
I20250813 19:59:22.302984 15813 raft_consensus.cc:357] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:22.305922 15813 raft_consensus.cc:738] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8e1adbb10c4e409605bdc9e95e29b2, State: Initialized, Role: FOLLOWER
I20250813 19:59:22.306258 15581 raft_consensus.cc:1273] T 52fe7343bf7f4f99a71f65498175f6a4 P af1510fe91334bcd8270e3ea9295b2db [term 3 FOLLOWER]: Refusing update from remote peer 75a403c4ca9b40a782c8146b272bcd9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250813 19:59:22.307186 15813 consensus_queue.cc:260] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } } peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:22.308230 15789 consensus_queue.cc:1035] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250813 19:59:22.311100 15813 ts_tablet_manager.cc:1428] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent starting tablet: real 0.028s user 0.027s sys 0.000s
I20250813 19:59:22.311774 15813 tablet_bootstrap.cc:492] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap starting.
I20250813 19:59:22.315063 15580 raft_consensus.cc:1273] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 3 FOLLOWER]: Refusing update from remote peer 75a403c4ca9b40a782c8146b272bcd9e: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250813 19:59:22.316742 15789 consensus_queue.cc:1035] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
W20250813 19:59:22.329813 15655 consensus_peers.cc:489] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e -> Peer aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): Couldn't send request to peer aa8e1adbb10c4e409605bdc9e95e29b2. Status: Network error: Client connection negotiation failed: client connection to 127.11.36.195:42161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250813 19:59:22.385468 15655 consensus_peers.cc:489] T 52fe7343bf7f4f99a71f65498175f6a4 P 75a403c4ca9b40a782c8146b272bcd9e -> Peer aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): Couldn't send request to peer aa8e1adbb10c4e409605bdc9e95e29b2. Status: Network error: Client connection negotiation failed: client connection to 127.11.36.195:42161: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250813 19:59:22.442768 15726 consensus_queue.cc:237] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } }
I20250813 19:59:22.444033 15813 tablet_bootstrap.cc:492] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:22.444875 15813 tablet_bootstrap.cc:492] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap complete.
I20250813 19:59:22.446444 15813 ts_tablet_manager.cc:1397] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent bootstrapping tablet: real 0.135s user 0.103s sys 0.027s
I20250813 19:59:22.448765 15813 raft_consensus.cc:357] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:59:22.448993 15580 raft_consensus.cc:1273] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 3 FOLLOWER]: Refusing update from remote peer 75a403c4ca9b40a782c8146b272bcd9e: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250813 19:59:22.449371 15813 raft_consensus.cc:738] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8e1adbb10c4e409605bdc9e95e29b2, State: Initialized, Role: FOLLOWER
I20250813 19:59:22.450016 15813 consensus_queue.cc:260] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:59:22.450515 15813 raft_consensus.cc:397] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:59:22.450793 15813 raft_consensus.cc:491] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:59:22.451148 15813 raft_consensus.cc:3058] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Advancing to term 3
I20250813 19:59:22.451699 15919 consensus_queue.cc:1035] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250813 19:59:22.458925 15813 raft_consensus.cc:513] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:59:22.459692 15813 leader_election.cc:304] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aa8e1adbb10c4e409605bdc9e95e29b2; no voters:
I20250813 19:59:22.458370 15919 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } }
I20250813 19:59:22.460433 15910 raft_consensus.cc:2802] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 3 FOLLOWER]: Leader election won for term 3
I20250813 19:59:22.461248 15580 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } }
I20250813 19:59:22.466354 15813 leader_election.cc:290] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250813 19:59:22.478330 15813 ts_tablet_manager.cc:1428] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent starting tablet: real 0.032s user 0.015s sys 0.004s
I20250813 19:59:22.479440 15813 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap starting.
I20250813 19:59:22.481544 15910 raft_consensus.cc:695] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 3 LEADER]: Becoming Leader. State: Replica: aa8e1adbb10c4e409605bdc9e95e29b2, State: Running, Role: LEADER
I20250813 19:59:22.482358 15910 consensus_queue.cc:237] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } }
I20250813 19:59:22.483114 15424 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 813334b0164e47a4aec2a01ec215d70f with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250813 19:59:22.489064 15438 catalog_manager.cc:5582] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: config changed from index 13 to 15, VOTER aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195) evicted. New cstate: current_term: 3 leader_uuid: "75a403c4ca9b40a782c8146b272bcd9e" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250813 19:59:22.513931 15438 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 813334b0164e47a4aec2a01ec215d70f on TS aa8e1adbb10c4e409605bdc9e95e29b2: Not found: failed to reset TS proxy: Could not find TS for UUID aa8e1adbb10c4e409605bdc9e95e29b2
I20250813 19:59:22.519675 15726 consensus_queue.cc:237] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } }
I20250813 19:59:22.523613 15919 raft_consensus.cc:2953] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER af1510fe91334bcd8270e3ea9295b2db (127.11.36.193) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } }
I20250813 19:59:22.531930 15424 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 813334b0164e47a4aec2a01ec215d70f with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250813 19:59:22.534770 15776 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.195:42161
I20250813 19:59:22.535039 15939 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.195:42161 every 8 connection(s)
I20250813 19:59:22.537031 15439 catalog_manager.cc:5582] T 813334b0164e47a4aec2a01ec215d70f P 75a403c4ca9b40a782c8146b272bcd9e reported cstate change: config changed from index 15 to 16, VOTER af1510fe91334bcd8270e3ea9295b2db (127.11.36.193) evicted. New cstate: current_term: 3 leader_uuid: "75a403c4ca9b40a782c8146b272bcd9e" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } health_report { overall_health: HEALTHY } } }
I20250813 19:59:22.538117 15776 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 19:59:22.543104 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 15776
W20250813 19:59:22.569862 15424 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 813334b0164e47a4aec2a01ec215d70f on TS aa8e1adbb10c4e409605bdc9e95e29b2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID aa8e1adbb10c4e409605bdc9e95e29b2
I20250813 19:59:22.590997 15561 tablet_service.cc:1515] Processing DeleteTablet for tablet 813334b0164e47a4aec2a01ec215d70f with delete_type TABLET_DATA_TOMBSTONED (TS af1510fe91334bcd8270e3ea9295b2db not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:35162
I20250813 19:59:22.594947 15940 heartbeater.cc:344] Connected to a master server at 127.11.36.254:40459
I20250813 19:59:22.595428 15940 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:22.596608 15940 heartbeater.cc:507] Master 127.11.36.254:40459 requested a full tablet report, sending...
I20250813 19:59:22.596771 15948 tablet_replica.cc:331] stopping tablet replica
I20250813 19:59:22.598417 15948 raft_consensus.cc:2241] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 3 FOLLOWER]: Raft consensus shutting down.
I20250813 19:59:22.599035 15948 raft_consensus.cc:2270] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db [term 3 FOLLOWER]: Raft consensus is shut down!
I20250813 19:59:22.600863 15438 ts_manager.cc:194] Registered new tserver with Master: aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161)
I20250813 19:59:22.601982 15948 ts_tablet_manager.cc:1905] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250813 19:59:22.602739 15438 catalog_manager.cc:5582] T 32c153406cdc460d9816ab3f39f5a7f6 P aa8e1adbb10c4e409605bdc9e95e29b2 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } health_report { overall_health: HEALTHY } } }
I20250813 19:59:22.610247 15438 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.195:41739
I20250813 19:59:22.612557 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 19:59:22.613893 15940 heartbeater.cc:499] Master 127.11.36.254:40459 was elected leader, sending a full tablet report...
I20250813 19:59:22.617113 15948 ts_tablet_manager.cc:1918] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250813 19:59:22.617496 15948 log.cc:1199] T 813334b0164e47a4aec2a01ec215d70f P af1510fe91334bcd8270e3ea9295b2db: Deleting WAL directory at /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/wals/813334b0164e47a4aec2a01ec215d70f
I20250813 19:59:22.619045 15423 catalog_manager.cc:4928] TS af1510fe91334bcd8270e3ea9295b2db (127.11.36.193:42107): tablet 813334b0164e47a4aec2a01ec215d70f (table TestTable [id=cfcb0a4a990942dcb80355dd990df81f]) successfully deleted
I20250813 19:59:22.621244 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250813 19:59:22.625672 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250813 19:59:22.633991 15856 tablet_service.cc:1515] Processing DeleteTablet for tablet 813334b0164e47a4aec2a01ec215d70f with delete_type TABLET_DATA_TOMBSTONED (TS aa8e1adbb10c4e409605bdc9e95e29b2 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:45984
W20250813 19:59:22.640960 15425 catalog_manager.cc:4908] TS aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): delete failed for tablet 813334b0164e47a4aec2a01ec215d70f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 813334b0164e47a4aec2a01ec215d70f already in progress: opening tablet
I20250813 19:59:22.656270 15813 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 19:59:22.656948 15813 tablet_bootstrap.cc:492] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Bootstrap complete.
I20250813 19:59:22.658131 15813 ts_tablet_manager.cc:1397] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent bootstrapping tablet: real 0.179s user 0.143s sys 0.010s
I20250813 19:59:22.659566 15813 raft_consensus.cc:357] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:22.659960 15813 raft_consensus.cc:738] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa8e1adbb10c4e409605bdc9e95e29b2, State: Initialized, Role: FOLLOWER
I20250813 19:59:22.660406 15813 consensus_queue.cc:260] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "75a403c4ca9b40a782c8146b272bcd9e" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43855 } } peers { permanent_uuid: "af1510fe91334bcd8270e3ea9295b2db" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42107 } attrs { promote: false } } peers { permanent_uuid: "aa8e1adbb10c4e409605bdc9e95e29b2" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 42161 } attrs { promote: false } }
I20250813 19:59:22.661520 15813 ts_tablet_manager.cc:1428] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250813 19:59:22.665426 15856 tablet_service.cc:1515] Processing DeleteTablet for tablet 813334b0164e47a4aec2a01ec215d70f with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 13 (current committed config index is 16)) from {username='slave'} at 127.0.0.1:45984
I20250813 19:59:22.666281 15950 tablet_replica.cc:331] stopping tablet replica
I20250813 19:59:22.666975 15950 raft_consensus.cc:2241] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250813 19:59:22.667450 15950 raft_consensus.cc:2270] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250813 19:59:22.669780 15950 ts_tablet_manager.cc:1905] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250813 19:59:22.679311 15950 ts_tablet_manager.cc:1918] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.13
I20250813 19:59:22.679625 15950 log.cc:1199] T 813334b0164e47a4aec2a01ec215d70f P aa8e1adbb10c4e409605bdc9e95e29b2: Deleting WAL directory at /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/wals/813334b0164e47a4aec2a01ec215d70f
I20250813 19:59:22.680784 15425 catalog_manager.cc:4928] TS aa8e1adbb10c4e409605bdc9e95e29b2 (127.11.36.195:42161): tablet 813334b0164e47a4aec2a01ec215d70f (table TestTable [id=cfcb0a4a990942dcb80355dd990df81f]) successfully deleted
I20250813 19:59:22.783324 15876 raft_consensus.cc:3058] T 52fe7343bf7f4f99a71f65498175f6a4 P aa8e1adbb10c4e409605bdc9e95e29b2 [term 2 FOLLOWER]: Advancing to term 3
W20250813 19:59:23.629410 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:24.632475 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:25.636006 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:26.639210 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:27.642146 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:28.645309 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:29.648393 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:30.651427 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:31.654707 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:32.658756 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:33.662148 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:34.665079 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:35.668354 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:36.671365 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:37.674263 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:38.677321 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:39.680361 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:40.683436 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250813 19:59:41.686623 11411 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 813334b0164e47a4aec2a01ec215d70f: tablet_id: "813334b0164e47a4aec2a01ec215d70f" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250813 19:59:42.690316 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 15476
I20250813 19:59:42.716624 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 15631
I20250813 19:59:42.743876 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 15776
I20250813 19:59:42.769918 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 15406
2025-08-13T19:59:42Z chronyd exiting
I20250813 19:59:42.818421 11411 test_util.cc:183] -----------------------------------------------
I20250813 19:59:42.818650 11411 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755115061406653-11411-0
[ FAILED ] AdminCliTest.TestRebuildTables (57552 ms)
[----------] 5 tests from AdminCliTest (121341 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250813 19:59:42.822501 11411 test_util.cc:276] Using random seed: -1727851441
I20250813 19:59:42.826597 11411 ts_itest-base.cc:115] Starting cluster with:
I20250813 19:59:42.826769 11411 ts_itest-base.cc:116] --------------
I20250813 19:59:42.826931 11411 ts_itest-base.cc:117] 5 tablet servers
I20250813 19:59:42.827070 11411 ts_itest-base.cc:118] 3 replicas per TS
I20250813 19:59:42.827199 11411 ts_itest-base.cc:119] --------------
2025-08-13T19:59:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T19:59:42Z Disabled control of system clock
I20250813 19:59:42.869065 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:44849
--webserver_interface=127.11.36.254
--webserver_port=0
--builtin_ntp_servers=127.11.36.212:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:44849
--raft_prepare_replacement_before_eviction=true with env {}
W20250813 19:59:43.148041 15972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:43.148608 15972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:43.149026 15972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:43.177449 15972 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 19:59:43.177785 15972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 19:59:43.177992 15972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:43.178181 15972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 19:59:43.178366 15972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 19:59:43.209465 15972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:44849
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:44849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:43.210654 15972 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:43.212010 15972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:43.221536 15978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:43.226061 15981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:43.222887 15979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:44.356781 15980 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1132 milliseconds
I20250813 19:59:44.356879 15972 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:44.358009 15972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:44.361621 15972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:44.363022 15972 hybrid_clock.cc:648] HybridClock initialized: now 1755115184362984 us; error 59 us; skew 500 ppm
I20250813 19:59:44.363699 15972 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:44.369719 15972 webserver.cc:480] Webserver started at http://127.11.36.254:42041/ using document root <none> and password file <none>
I20250813 19:59:44.370606 15972 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:44.370792 15972 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:44.371284 15972 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:59:44.375241 15972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9b9e41cd8b22488b875af352b63fdfa9"
format_stamp: "Formatted at 2025-08-13 19:59:44 on dist-test-slave-l098"
I20250813 19:59:44.376250 15972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9b9e41cd8b22488b875af352b63fdfa9"
format_stamp: "Formatted at 2025-08-13 19:59:44 on dist-test-slave-l098"
I20250813 19:59:44.382911 15972 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20250813 19:59:44.388099 15988 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:44.389043 15972 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250813 19:59:44.389359 15972 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9b9e41cd8b22488b875af352b63fdfa9"
format_stamp: "Formatted at 2025-08-13 19:59:44 on dist-test-slave-l098"
I20250813 19:59:44.389657 15972 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:44.438151 15972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:44.439522 15972 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:44.439921 15972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:44.505753 15972 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:44849
I20250813 19:59:44.505815 16039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:44849 every 8 connection(s)
I20250813 19:59:44.508332 15972 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 19:59:44.513049 16040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:59:44.514549 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 15972
I20250813 19:59:44.514942 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 19:59:44.534590 16040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9: Bootstrap starting.
I20250813 19:59:44.539454 16040 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9: Neither blocks nor log segments found. Creating new log.
I20250813 19:59:44.541415 16040 log.cc:826] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:44.546656 16040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9: No bootstrap required, opened a new log
I20250813 19:59:44.563287 16040 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9e41cd8b22488b875af352b63fdfa9" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 44849 } }
I20250813 19:59:44.563868 16040 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:59:44.564056 16040 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9e41cd8b22488b875af352b63fdfa9, State: Initialized, Role: FOLLOWER
I20250813 19:59:44.564728 16040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9e41cd8b22488b875af352b63fdfa9" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 44849 } }
I20250813 19:59:44.565320 16040 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 19:59:44.565579 16040 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 19:59:44.565896 16040 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:59:44.569878 16040 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9e41cd8b22488b875af352b63fdfa9" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 44849 } }
I20250813 19:59:44.570614 16040 leader_election.cc:304] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b9e41cd8b22488b875af352b63fdfa9; no voters:
I20250813 19:59:44.572427 16040 leader_election.cc:290] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 19:59:44.573101 16045 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:59:44.575119 16045 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [term 1 LEADER]: Becoming Leader. State: Replica: 9b9e41cd8b22488b875af352b63fdfa9, State: Running, Role: LEADER
I20250813 19:59:44.575834 16045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9e41cd8b22488b875af352b63fdfa9" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 44849 } }
I20250813 19:59:44.576761 16040 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 19:59:44.585851 16047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b9e41cd8b22488b875af352b63fdfa9. Latest consensus state: current_term: 1 leader_uuid: "9b9e41cd8b22488b875af352b63fdfa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9e41cd8b22488b875af352b63fdfa9" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 44849 } } }
I20250813 19:59:44.586364 16047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [sys.catalog]: This master's current role is: LEADER
I20250813 19:59:44.586649 16046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b9e41cd8b22488b875af352b63fdfa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b9e41cd8b22488b875af352b63fdfa9" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 44849 } } }
I20250813 19:59:44.587430 16046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9 [sys.catalog]: This master's current role is: LEADER
I20250813 19:59:44.589169 16053 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 19:59:44.599974 16053 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 19:59:44.616466 16053 catalog_manager.cc:1349] Generated new cluster ID: ce0c580face947eba9ffacfda0ff707b
I20250813 19:59:44.616729 16053 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 19:59:44.646842 16053 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 19:59:44.648692 16053 catalog_manager.cc:1506] Loading token signing keys...
I20250813 19:59:44.664498 16053 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9b9e41cd8b22488b875af352b63fdfa9: Generated new TSK 0
I20250813 19:59:44.665333 16053 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 19:59:44.681911 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:0
--local_ip_for_outbound_sockets=127.11.36.193
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:44849
--builtin_ntp_servers=127.11.36.212:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250813 19:59:44.976171 16064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:44.976656 16064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:44.977099 16064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:45.006160 16064 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 19:59:45.006579 16064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:45.007349 16064 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 19:59:45.040433 16064 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:44849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:45.041682 16064 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:45.043288 16064 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:45.055624 16070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:45.059902 16073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:45.058320 16071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:46.421054 16072 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1360 milliseconds
I20250813 19:59:46.421164 16064 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:46.422430 16064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:46.424983 16064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:46.426431 16064 hybrid_clock.cc:648] HybridClock initialized: now 1755115186426378 us; error 80 us; skew 500 ppm
I20250813 19:59:46.427150 16064 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:46.433908 16064 webserver.cc:480] Webserver started at http://127.11.36.193:44907/ using document root <none> and password file <none>
I20250813 19:59:46.434782 16064 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:46.434988 16064 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:46.435413 16064 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:59:46.439496 16064 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "eab931652f65459b868835e60ef3cfd1"
format_stamp: "Formatted at 2025-08-13 19:59:46 on dist-test-slave-l098"
I20250813 19:59:46.440822 16064 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "eab931652f65459b868835e60ef3cfd1"
format_stamp: "Formatted at 2025-08-13 19:59:46 on dist-test-slave-l098"
I20250813 19:59:46.448024 16064 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20250813 19:59:46.454373 16080 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:46.455497 16064 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250813 19:59:46.455791 16064 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "eab931652f65459b868835e60ef3cfd1"
format_stamp: "Formatted at 2025-08-13 19:59:46 on dist-test-slave-l098"
I20250813 19:59:46.456126 16064 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:46.502990 16064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:46.504848 16064 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:46.505348 16064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:46.508410 16064 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:46.513782 16064 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:59:46.514055 16064 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:46.514331 16064 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:59:46.514533 16064 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:46.684665 16064 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:44813
I20250813 19:59:46.684820 16192 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:44813 every 8 connection(s)
I20250813 19:59:46.687357 16064 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 19:59:46.694528 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 16064
I20250813 19:59:46.695057 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 19:59:46.701498 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.194:0
--local_ip_for_outbound_sockets=127.11.36.194
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:44849
--builtin_ntp_servers=127.11.36.212:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250813 19:59:46.714109 16193 heartbeater.cc:344] Connected to a master server at 127.11.36.254:44849
I20250813 19:59:46.714663 16193 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:46.716035 16193 heartbeater.cc:507] Master 127.11.36.254:44849 requested a full tablet report, sending...
I20250813 19:59:46.719070 16005 ts_manager.cc:194] Registered new tserver with Master: eab931652f65459b868835e60ef3cfd1 (127.11.36.193:44813)
I20250813 19:59:46.721727 16005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:52889
W20250813 19:59:47.008885 16197 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:47.009418 16197 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:47.009907 16197 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:47.041795 16197 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 19:59:47.042217 16197 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:47.042999 16197 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.194
I20250813 19:59:47.078049 16197 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:44849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.194
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:47.079382 16197 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:47.080960 16197 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:47.093611 16203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:59:47.725903 16193 heartbeater.cc:499] Master 127.11.36.254:44849 was elected leader, sending a full tablet report...
W20250813 19:59:47.094878 16204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:48.226555 16206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:48.228683 16205 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1131 milliseconds
I20250813 19:59:48.228785 16197 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:48.229976 16197 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:48.232025 16197 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:48.233381 16197 hybrid_clock.cc:648] HybridClock initialized: now 1755115188233310 us; error 81 us; skew 500 ppm
I20250813 19:59:48.234108 16197 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:48.239785 16197 webserver.cc:480] Webserver started at http://127.11.36.194:42227/ using document root <none> and password file <none>
I20250813 19:59:48.240672 16197 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:48.240865 16197 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:48.241293 16197 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:59:48.245366 16197 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "309e29a128c4499685c1236985474c8d"
format_stamp: "Formatted at 2025-08-13 19:59:48 on dist-test-slave-l098"
I20250813 19:59:48.246374 16197 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "309e29a128c4499685c1236985474c8d"
format_stamp: "Formatted at 2025-08-13 19:59:48 on dist-test-slave-l098"
I20250813 19:59:48.252948 16197 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250813 19:59:48.258248 16213 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:48.259198 16197 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250813 19:59:48.259487 16197 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "309e29a128c4499685c1236985474c8d"
format_stamp: "Formatted at 2025-08-13 19:59:48 on dist-test-slave-l098"
I20250813 19:59:48.259790 16197 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:48.317269 16197 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:48.318637 16197 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:48.319028 16197 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:48.321544 16197 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:48.325444 16197 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:59:48.325625 16197 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:48.325879 16197 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:59:48.326021 16197 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:48.452267 16197 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.194:43785
I20250813 19:59:48.452350 16325 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.194:43785 every 8 connection(s)
I20250813 19:59:48.454741 16197 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250813 19:59:48.465129 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 16197
I20250813 19:59:48.465520 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250813 19:59:48.471449 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.195:0
--local_ip_for_outbound_sockets=127.11.36.195
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:44849
--builtin_ntp_servers=127.11.36.212:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250813 19:59:48.474004 16326 heartbeater.cc:344] Connected to a master server at 127.11.36.254:44849
I20250813 19:59:48.474504 16326 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:48.475800 16326 heartbeater.cc:507] Master 127.11.36.254:44849 requested a full tablet report, sending...
I20250813 19:59:48.478127 16005 ts_manager.cc:194] Registered new tserver with Master: 309e29a128c4499685c1236985474c8d (127.11.36.194:43785)
I20250813 19:59:48.479835 16005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.194:35497
W20250813 19:59:48.755879 16330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:48.756492 16330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:48.756973 16330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:48.785897 16330 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 19:59:48.786275 16330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:48.786989 16330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.195
I20250813 19:59:48.819103 16330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:44849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.195
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:48.820396 16330 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:48.821911 16330 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:48.833194 16336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:59:49.482976 16326 heartbeater.cc:499] Master 127.11.36.254:44849 was elected leader, sending a full tablet report...
W20250813 19:59:48.833725 16337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:49.945273 16339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:49.947678 16338 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1110 milliseconds
I20250813 19:59:49.947831 16330 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:49.949082 16330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:49.951690 16330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:49.953091 16330 hybrid_clock.cc:648] HybridClock initialized: now 1755115189953053 us; error 48 us; skew 500 ppm
I20250813 19:59:49.953810 16330 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:49.960129 16330 webserver.cc:480] Webserver started at http://127.11.36.195:45959/ using document root <none> and password file <none>
I20250813 19:59:49.961036 16330 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:49.961210 16330 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:49.961608 16330 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:59:49.965878 16330 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "24a59082c135485c82a8dea1edbae9da"
format_stamp: "Formatted at 2025-08-13 19:59:49 on dist-test-slave-l098"
I20250813 19:59:49.966818 16330 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "24a59082c135485c82a8dea1edbae9da"
format_stamp: "Formatted at 2025-08-13 19:59:49 on dist-test-slave-l098"
I20250813 19:59:49.973446 16330 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20250813 19:59:49.979020 16346 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:49.979954 16330 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250813 19:59:49.980252 16330 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "24a59082c135485c82a8dea1edbae9da"
format_stamp: "Formatted at 2025-08-13 19:59:49 on dist-test-slave-l098"
I20250813 19:59:49.980561 16330 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:50.029178 16330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:50.030540 16330 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:50.030946 16330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:50.033169 16330 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:50.036888 16330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:59:50.037056 16330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:50.037218 16330 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:59:50.037330 16330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:50.163167 16330 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.195:43123
I20250813 19:59:50.163267 16458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.195:43123 every 8 connection(s)
I20250813 19:59:50.165591 16330 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250813 19:59:50.172004 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 16330
I20250813 19:59:50.172407 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250813 19:59:50.178401 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.196:0
--local_ip_for_outbound_sockets=127.11.36.196
--webserver_interface=127.11.36.196
--webserver_port=0
--tserver_master_addrs=127.11.36.254:44849
--builtin_ntp_servers=127.11.36.212:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250813 19:59:50.189715 16459 heartbeater.cc:344] Connected to a master server at 127.11.36.254:44849
I20250813 19:59:50.190130 16459 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:50.191102 16459 heartbeater.cc:507] Master 127.11.36.254:44849 requested a full tablet report, sending...
I20250813 19:59:50.193225 16005 ts_manager.cc:194] Registered new tserver with Master: 24a59082c135485c82a8dea1edbae9da (127.11.36.195:43123)
I20250813 19:59:50.194372 16005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.195:36887
W20250813 19:59:50.462224 16463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:50.462720 16463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:50.463191 16463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:50.494837 16463 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 19:59:50.495218 16463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:50.495906 16463 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.196
I20250813 19:59:50.530993 16463 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.11.36.196
--webserver_port=0
--tserver_master_addrs=127.11.36.254:44849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.196
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:50.532250 16463 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:50.533762 16463 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:50.544872 16469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:59:51.197324 16459 heartbeater.cc:499] Master 127.11.36.254:44849 was elected leader, sending a full tablet report...
W20250813 19:59:51.948311 16468 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 16463
W20250813 19:59:52.156755 16463 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.612s user 0.670s sys 0.941s
W20250813 19:59:50.545339 16470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:52.157109 16463 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.613s user 0.670s sys 0.941s
W20250813 19:59:52.159045 16472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:52.161310 16471 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1612 milliseconds
I20250813 19:59:52.161340 16463 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:52.162711 16463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:52.165210 16463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:52.166684 16463 hybrid_clock.cc:648] HybridClock initialized: now 1755115192166636 us; error 47 us; skew 500 ppm
I20250813 19:59:52.167649 16463 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:52.174748 16463 webserver.cc:480] Webserver started at http://127.11.36.196:41513/ using document root <none> and password file <none>
I20250813 19:59:52.175936 16463 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:52.176216 16463 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:52.176738 16463 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:59:52.180707 16463 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "a0e4ff3f1d414961ac13a93c8bb52ecf"
format_stamp: "Formatted at 2025-08-13 19:59:52 on dist-test-slave-l098"
I20250813 19:59:52.181699 16463 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "a0e4ff3f1d414961ac13a93c8bb52ecf"
format_stamp: "Formatted at 2025-08-13 19:59:52 on dist-test-slave-l098"
I20250813 19:59:52.187945 16463 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20250813 19:59:52.193331 16480 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:52.194239 16463 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250813 19:59:52.194523 16463 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "a0e4ff3f1d414961ac13a93c8bb52ecf"
format_stamp: "Formatted at 2025-08-13 19:59:52 on dist-test-slave-l098"
I20250813 19:59:52.194800 16463 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:52.240386 16463 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:52.241706 16463 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:52.242080 16463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:52.244474 16463 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:52.248059 16463 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:59:52.248302 16463 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250813 19:59:52.248548 16463 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:59:52.248687 16463 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:52.371126 16463 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.196:40299
I20250813 19:59:52.371214 16592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.196:40299 every 8 connection(s)
I20250813 19:59:52.373483 16463 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250813 19:59:52.376263 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 16463
I20250813 19:59:52.376935 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250813 19:59:52.384063 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.197:0
--local_ip_for_outbound_sockets=127.11.36.197
--webserver_interface=127.11.36.197
--webserver_port=0
--tserver_master_addrs=127.11.36.254:44849
--builtin_ntp_servers=127.11.36.212:41951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250813 19:59:52.393610 16593 heartbeater.cc:344] Connected to a master server at 127.11.36.254:44849
I20250813 19:59:52.394038 16593 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:52.394968 16593 heartbeater.cc:507] Master 127.11.36.254:44849 requested a full tablet report, sending...
I20250813 19:59:52.396898 16005 ts_manager.cc:194] Registered new tserver with Master: a0e4ff3f1d414961ac13a93c8bb52ecf (127.11.36.196:40299)
I20250813 19:59:52.398065 16005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.196:50343
W20250813 19:59:52.676947 16597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 19:59:52.677446 16597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 19:59:52.677956 16597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 19:59:52.707679 16597 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250813 19:59:52.708019 16597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 19:59:52.708794 16597 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.197
I20250813 19:59:52.740553 16597 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:41951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.11.36.197
--webserver_port=0
--tserver_master_addrs=127.11.36.254:44849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.197
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 19:59:52.741770 16597 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 19:59:52.743132 16597 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 19:59:52.753844 16603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 19:59:53.400899 16593 heartbeater.cc:499] Master 127.11.36.254:44849 was elected leader, sending a full tablet report...
W20250813 19:59:52.755321 16604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:53.867377 16606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 19:59:53.869134 16605 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1110 milliseconds
I20250813 19:59:53.869215 16597 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 19:59:53.870357 16597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 19:59:53.872864 16597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 19:59:53.874255 16597 hybrid_clock.cc:648] HybridClock initialized: now 1755115193874209 us; error 51 us; skew 500 ppm
I20250813 19:59:53.874982 16597 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 19:59:53.880932 16597 webserver.cc:480] Webserver started at http://127.11.36.197:38087/ using document root <none> and password file <none>
I20250813 19:59:53.881812 16597 fs_manager.cc:362] Metadata directory not provided
I20250813 19:59:53.882042 16597 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 19:59:53.882452 16597 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 19:59:53.886381 16597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "6a47b0fa5ee140c792f5ddef0065eb17"
format_stamp: "Formatted at 2025-08-13 19:59:53 on dist-test-slave-l098"
I20250813 19:59:53.887302 16597 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "6a47b0fa5ee140c792f5ddef0065eb17"
format_stamp: "Formatted at 2025-08-13 19:59:53 on dist-test-slave-l098"
I20250813 19:59:53.894181 16597 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250813 19:59:53.899271 16614 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:53.900290 16597 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20250813 19:59:53.900564 16597 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "6a47b0fa5ee140c792f5ddef0065eb17"
format_stamp: "Formatted at 2025-08-13 19:59:53 on dist-test-slave-l098"
I20250813 19:59:53.900877 16597 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 19:59:53.943100 16597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 19:59:53.944484 16597 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 19:59:53.944892 16597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 19:59:53.947115 16597 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 19:59:53.950760 16597 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 19:59:53.950937 16597 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:53.951238 16597 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 19:59:53.951377 16597 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 19:59:54.085649 16597 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.197:33859
I20250813 19:59:54.085740 16726 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.197:33859 every 8 connection(s)
I20250813 19:59:54.087951 16597 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250813 19:59:54.098436 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 16597
I20250813 19:59:54.098796 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250813 19:59:54.109378 16727 heartbeater.cc:344] Connected to a master server at 127.11.36.254:44849
I20250813 19:59:54.109743 16727 heartbeater.cc:461] Registering TS with master...
I20250813 19:59:54.110670 16727 heartbeater.cc:507] Master 127.11.36.254:44849 requested a full tablet report, sending...
I20250813 19:59:54.112550 16005 ts_manager.cc:194] Registered new tserver with Master: 6a47b0fa5ee140c792f5ddef0065eb17 (127.11.36.197:33859)
I20250813 19:59:54.113688 16005 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.197:47839
I20250813 19:59:54.119261 11411 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250813 19:59:54.152616 16005 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58644:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250813 19:59:54.231423 16261 tablet_service.cc:1468] Processing CreateTablet for tablet 3da52b6ad1164c50b1c672ba7ed63e10 (DEFAULT_TABLE table=TestTable [id=8c5efc0203294f0ba29525f1822637bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:59:54.233341 16261 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da52b6ad1164c50b1c672ba7ed63e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:59:54.234320 16662 tablet_service.cc:1468] Processing CreateTablet for tablet 3da52b6ad1164c50b1c672ba7ed63e10 (DEFAULT_TABLE table=TestTable [id=8c5efc0203294f0ba29525f1822637bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:59:54.236466 16662 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da52b6ad1164c50b1c672ba7ed63e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:59:54.237645 16394 tablet_service.cc:1468] Processing CreateTablet for tablet 3da52b6ad1164c50b1c672ba7ed63e10 (DEFAULT_TABLE table=TestTable [id=8c5efc0203294f0ba29525f1822637bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 19:59:54.239876 16394 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3da52b6ad1164c50b1c672ba7ed63e10. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 19:59:54.254180 16746 tablet_bootstrap.cc:492] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d: Bootstrap starting.
I20250813 19:59:54.259630 16746 tablet_bootstrap.cc:654] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d: Neither blocks nor log segments found. Creating new log.
I20250813 19:59:54.261968 16746 log.cc:826] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:54.265096 16747 tablet_bootstrap.cc:492] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17: Bootstrap starting.
I20250813 19:59:54.267697 16749 tablet_bootstrap.cc:492] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da: Bootstrap starting.
I20250813 19:59:54.271950 16746 tablet_bootstrap.cc:492] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d: No bootstrap required, opened a new log
I20250813 19:59:54.272315 16747 tablet_bootstrap.cc:654] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17: Neither blocks nor log segments found. Creating new log.
I20250813 19:59:54.272455 16746 ts_tablet_manager.cc:1397] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.008s
I20250813 19:59:54.274307 16747 log.cc:826] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:54.275159 16749 tablet_bootstrap.cc:654] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da: Neither blocks nor log segments found. Creating new log.
I20250813 19:59:54.278976 16749 log.cc:826] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da: Log is configured to *not* fsync() on all Append() calls
I20250813 19:59:54.282054 16747 tablet_bootstrap.cc:492] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17: No bootstrap required, opened a new log
I20250813 19:59:54.282491 16747 ts_tablet_manager.cc:1397] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.008s
I20250813 19:59:54.284355 16749 tablet_bootstrap.cc:492] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da: No bootstrap required, opened a new log
I20250813 19:59:54.284773 16749 ts_tablet_manager.cc:1397] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.006s
I20250813 19:59:54.298038 16746 raft_consensus.cc:357] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 } } peers { permanent_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" member_type: VOTER last_known_addr { host: "127.11.36.197" port: 33859 } } peers { permanent_uuid: "24a59082c135485c82a8dea1edbae9da" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43123 } }
I20250813 19:59:54.298837 16746 raft_consensus.cc:383] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:59:54.299099 16746 raft_consensus.cc:738] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 309e29a128c4499685c1236985474c8d, State: Initialized, Role: FOLLOWER
I20250813 19:59:54.300122 16747 raft_consensus.cc:357] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 } } peers { permanent_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" member_type: VOTER last_known_addr { host: "127.11.36.197" port: 33859 } } peers { permanent_uuid: "24a59082c135485c82a8dea1edbae9da" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43123 } }
I20250813 19:59:54.299957 16746 consensus_queue.cc:260] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 } } peers { permanent_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" member_type: VOTER last_known_addr { host: "127.11.36.197" port: 33859 } } peers { permanent_uuid: "24a59082c135485c82a8dea1edbae9da" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43123 } }
I20250813 19:59:54.300714 16747 raft_consensus.cc:383] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:59:54.300978 16747 raft_consensus.cc:738] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a47b0fa5ee140c792f5ddef0065eb17, State: Initialized, Role: FOLLOWER
I20250813 19:59:54.301874 16747 consensus_queue.cc:260] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 } } peers { permanent_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" member_type: VOTER last_known_addr { host: "127.11.36.197" port: 33859 } } peers { permanent_uuid: "24a59082c135485c82a8dea1edbae9da" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43123 } }
I20250813 19:59:54.304366 16746 ts_tablet_manager.cc:1428] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d: Time spent starting tablet: real 0.032s user 0.031s sys 0.001s
I20250813 19:59:54.309515 16727 heartbeater.cc:499] Master 127.11.36.254:44849 was elected leader, sending a full tablet report...
I20250813 19:59:54.309088 16749 raft_consensus.cc:357] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 } } peers { permanent_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" member_type: VOTER last_known_addr { host: "127.11.36.197" port: 33859 } } peers { permanent_uuid: "24a59082c135485c82a8dea1edbae9da" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43123 } }
I20250813 19:59:54.309976 16749 raft_consensus.cc:383] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 19:59:54.310300 16749 raft_consensus.cc:738] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a59082c135485c82a8dea1edbae9da, State: Initialized, Role: FOLLOWER
I20250813 19:59:54.310776 16747 ts_tablet_manager.cc:1428] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17: Time spent starting tablet: real 0.028s user 0.023s sys 0.004s
I20250813 19:59:54.311225 16749 consensus_queue.cc:260] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 } } peers { permanent_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" member_type: VOTER last_known_addr { host: "127.11.36.197" port: 33859 } } peers { permanent_uuid: "24a59082c135485c82a8dea1edbae9da" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43123 } }
I20250813 19:59:54.315259 16749 ts_tablet_manager.cc:1428] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da: Time spent starting tablet: real 0.030s user 0.020s sys 0.010s
I20250813 19:59:54.323421 16754 raft_consensus.cc:491] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 19:59:54.323840 16754 raft_consensus.cc:513] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 } } peers { permanent_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" member_type: VOTER last_known_addr { host: "127.11.36.197" port: 33859 } } peers { permanent_uuid: "24a59082c135485c82a8dea1edbae9da" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43123 } }
I20250813 19:59:54.326392 16754 leader_election.cc:290] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 309e29a128c4499685c1236985474c8d (127.11.36.194:43785), 6a47b0fa5ee140c792f5ddef0065eb17 (127.11.36.197:33859)
I20250813 19:59:54.339996 16682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3da52b6ad1164c50b1c672ba7ed63e10" candidate_uuid: "24a59082c135485c82a8dea1edbae9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" is_pre_election: true
I20250813 19:59:54.340808 16682 raft_consensus.cc:2466] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a59082c135485c82a8dea1edbae9da in term 0.
I20250813 19:59:54.340683 16281 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3da52b6ad1164c50b1c672ba7ed63e10" candidate_uuid: "24a59082c135485c82a8dea1edbae9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309e29a128c4499685c1236985474c8d" is_pre_election: true
I20250813 19:59:54.341547 16281 raft_consensus.cc:2466] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24a59082c135485c82a8dea1edbae9da in term 0.
I20250813 19:59:54.341990 16350 leader_election.cc:304] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a59082c135485c82a8dea1edbae9da, 6a47b0fa5ee140c792f5ddef0065eb17; no voters:
I20250813 19:59:54.342756 16754 raft_consensus.cc:2802] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 19:59:54.343117 16754 raft_consensus.cc:491] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250813 19:59:54.343220 16728 tablet.cc:2378] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:59:54.343353 16754 raft_consensus.cc:3058] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:59:54.347841 16754 raft_consensus.cc:513] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 } } peers { permanent_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" member_type: VOTER last_known_addr { host: "127.11.36.197" port: 33859 } } peers { permanent_uuid: "24a59082c135485c82a8dea1edbae9da" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43123 } }
I20250813 19:59:54.349071 16754 leader_election.cc:290] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [CANDIDATE]: Term 1 election: Requested vote from peers 309e29a128c4499685c1236985474c8d (127.11.36.194:43785), 6a47b0fa5ee140c792f5ddef0065eb17 (127.11.36.197:33859)
I20250813 19:59:54.349737 16281 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3da52b6ad1164c50b1c672ba7ed63e10" candidate_uuid: "24a59082c135485c82a8dea1edbae9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "309e29a128c4499685c1236985474c8d"
I20250813 19:59:54.349828 16682 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3da52b6ad1164c50b1c672ba7ed63e10" candidate_uuid: "24a59082c135485c82a8dea1edbae9da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a47b0fa5ee140c792f5ddef0065eb17"
I20250813 19:59:54.350147 16281 raft_consensus.cc:3058] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:59:54.350270 16682 raft_consensus.cc:3058] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17 [term 0 FOLLOWER]: Advancing to term 1
I20250813 19:59:54.354724 16281 raft_consensus.cc:2466] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a59082c135485c82a8dea1edbae9da in term 1.
I20250813 19:59:54.355116 16682 raft_consensus.cc:2466] T 3da52b6ad1164c50b1c672ba7ed63e10 P 6a47b0fa5ee140c792f5ddef0065eb17 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24a59082c135485c82a8dea1edbae9da in term 1.
I20250813 19:59:54.355665 16349 leader_election.cc:304] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24a59082c135485c82a8dea1edbae9da, 309e29a128c4499685c1236985474c8d; no voters:
I20250813 19:59:54.356408 16754 raft_consensus.cc:2802] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 1 FOLLOWER]: Leader election won for term 1
I20250813 19:59:54.358168 16754 raft_consensus.cc:695] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [term 1 LEADER]: Becoming Leader. State: Replica: 24a59082c135485c82a8dea1edbae9da, State: Running, Role: LEADER
I20250813 19:59:54.358777 16754 consensus_queue.cc:237] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 } } peers { permanent_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" member_type: VOTER last_known_addr { host: "127.11.36.197" port: 33859 } } peers { permanent_uuid: "24a59082c135485c82a8dea1edbae9da" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43123 } }
I20250813 19:59:54.368854 16005 catalog_manager.cc:5582] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da reported cstate change: term changed from 0 to 1, leader changed from <none> to 24a59082c135485c82a8dea1edbae9da (127.11.36.195). New cstate: current_term: 1 leader_uuid: "24a59082c135485c82a8dea1edbae9da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6a47b0fa5ee140c792f5ddef0065eb17" member_type: VOTER last_known_addr { host: "127.11.36.197" port: 33859 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24a59082c135485c82a8dea1edbae9da" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 43123 } health_report { overall_health: HEALTHY } } }
I20250813 19:59:54.417562 11411 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250813 19:59:54.420467 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 309e29a128c4499685c1236985474c8d to finish bootstrapping
W20250813 19:59:54.424015 16460 tablet.cc:2378] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:59:54.432736 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 24a59082c135485c82a8dea1edbae9da to finish bootstrapping
I20250813 19:59:54.443058 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6a47b0fa5ee140c792f5ddef0065eb17 to finish bootstrapping
I20250813 19:59:54.452626 11411 test_util.cc:276] Using random seed: -1716221329
W20250813 19:59:54.465327 16327 tablet.cc:2378] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 19:59:54.477874 11411 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250813 19:59:54.478765 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 16597
W20250813 19:59:54.510960 16350 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.36.197:33859: connect: Connection refused (error 111)
I20250813 19:59:54.511706 16281 raft_consensus.cc:1273] T 3da52b6ad1164c50b1c672ba7ed63e10 P 309e29a128c4499685c1236985474c8d [term 1 FOLLOWER]: Refusing update from remote peer 24a59082c135485c82a8dea1edbae9da: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250813 19:59:54.513536 16757 consensus_queue.cc:1035] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da [LEADER]: Connected to new peer: Peer: permanent_uuid: "309e29a128c4499685c1236985474c8d" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 43785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250813 19:59:54.530351 16350 consensus_peers.cc:489] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da -> Peer 6a47b0fa5ee140c792f5ddef0065eb17 (127.11.36.197:33859): Couldn't send request to peer 6a47b0fa5ee140c792f5ddef0065eb17. Status: Network error: Client connection negotiation failed: client connection to 127.11.36.197:33859: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250813 19:59:54.545061 16767 mvcc.cc:204] Tried to move back new op lower bound from 7188951836692135936 to 7188951836099563520. Current Snapshot: MvccSnapshot[applied={T|T < 7188951836692135936}]
W20250813 19:59:57.121137 16350 consensus_peers.cc:489] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da -> Peer 6a47b0fa5ee140c792f5ddef0065eb17 (127.11.36.197:33859): Couldn't send request to peer 6a47b0fa5ee140c792f5ddef0065eb17. Status: Network error: Client connection negotiation failed: client connection to 127.11.36.197:33859: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250813 19:59:57.174801 16394 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 19:59:57.195343 16261 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 19:59:57.233179 16528 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 19:59:57.283023 16128 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 19:59:58.998510 16261 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 19:59:58.999336 16394 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 19:59:59.029583 16528 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 19:59:59.033586 16128 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250813 19:59:59.617707 16350 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.36.197:33859: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250813 19:59:59.619969 16350 consensus_peers.cc:489] T 3da52b6ad1164c50b1c672ba7ed63e10 P 24a59082c135485c82a8dea1edbae9da -> Peer 6a47b0fa5ee140c792f5ddef0065eb17 (127.11.36.197:33859): Couldn't send request to peer 6a47b0fa5ee140c792f5ddef0065eb17. Status: Network error: Client connection negotiation failed: client connection to 127.11.36.197:33859: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250813 20:00:01.437529 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 16064
I20250813 20:00:01.469923 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 16197
I20250813 20:00:01.503183 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 16330
I20250813 20:00:01.554306 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 16463
I20250813 20:00:01.578517 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 15972
2025-08-13T20:00:01Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (18813 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (18814 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250813 20:00:01.636703 11411 test_util.cc:276] Using random seed: -1709037249
I20250813 20:00:01.640640 11411 ts_itest-base.cc:115] Starting cluster with:
I20250813 20:00:01.640797 11411 ts_itest-base.cc:116] --------------
I20250813 20:00:01.640954 11411 ts_itest-base.cc:117] 1 tablet servers
I20250813 20:00:01.641101 11411 ts_itest-base.cc:118] 1 replicas per TS
I20250813 20:00:01.641240 11411 ts_itest-base.cc:119] --------------
2025-08-13T20:00:01Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T20:00:01Z Disabled control of system clock
I20250813 20:00:01.683624 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:41603
--webserver_interface=127.11.36.254
--webserver_port=0
--builtin_ntp_servers=127.11.36.212:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:41603 with env {}
W20250813 20:00:01.978423 16870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:01.979008 16870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 20:00:01.979440 16870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:02.007731 16870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 20:00:02.008005 16870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 20:00:02.008271 16870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 20:00:02.008482 16870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 20:00:02.041460 16870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:41603
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 20:00:02.042675 16870 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:02.044152 16870 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:02.053797 16877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:03.458175 16876 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 16870
W20250813 20:00:03.731675 16870 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.678s user 0.575s sys 1.101s
W20250813 20:00:03.732214 16870 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.678s user 0.576s sys 1.101s
W20250813 20:00:02.054395 16878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:03.733899 16880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:03.736272 16879 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1678 milliseconds
I20250813 20:00:03.736361 16870 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 20:00:03.737550 16870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 20:00:03.739903 16870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 20:00:03.741238 16870 hybrid_clock.cc:648] HybridClock initialized: now 1755115203741193 us; error 72 us; skew 500 ppm
I20250813 20:00:03.741990 16870 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:03.747432 16870 webserver.cc:480] Webserver started at http://127.11.36.254:41515/ using document root <none> and password file <none>
I20250813 20:00:03.748302 16870 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:03.748499 16870 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:03.748930 16870 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 20:00:03.754504 16870 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9f59ecd4556948428ba4957093aea8ba"
format_stamp: "Formatted at 2025-08-13 20:00:03 on dist-test-slave-l098"
I20250813 20:00:03.755556 16870 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9f59ecd4556948428ba4957093aea8ba"
format_stamp: "Formatted at 2025-08-13 20:00:03 on dist-test-slave-l098"
I20250813 20:00:03.762147 16870 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20250813 20:00:03.767277 16887 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:03.768260 16870 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250813 20:00:03.768532 16870 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9f59ecd4556948428ba4957093aea8ba"
format_stamp: "Formatted at 2025-08-13 20:00:03 on dist-test-slave-l098"
I20250813 20:00:03.768843 16870 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:03.826820 16870 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:03.828289 16870 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:03.828716 16870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:03.895481 16870 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:41603
I20250813 20:00:03.895519 16938 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:41603 every 8 connection(s)
I20250813 20:00:03.898203 16870 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250813 20:00:03.903147 16939 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 20:00:03.908313 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 16870
I20250813 20:00:03.908831 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250813 20:00:03.926555 16939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba: Bootstrap starting.
I20250813 20:00:03.931679 16939 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba: Neither blocks nor log segments found. Creating new log.
I20250813 20:00:03.933419 16939 log.cc:826] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba: Log is configured to *not* fsync() on all Append() calls
I20250813 20:00:03.937903 16939 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba: No bootstrap required, opened a new log
I20250813 20:00:03.954272 16939 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59ecd4556948428ba4957093aea8ba" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 41603 } }
I20250813 20:00:03.954921 16939 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 20:00:03.955184 16939 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f59ecd4556948428ba4957093aea8ba, State: Initialized, Role: FOLLOWER
I20250813 20:00:03.955859 16939 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59ecd4556948428ba4957093aea8ba" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 41603 } }
I20250813 20:00:03.956400 16939 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 20:00:03.956647 16939 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 20:00:03.956950 16939 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [term 0 FOLLOWER]: Advancing to term 1
I20250813 20:00:03.960994 16939 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59ecd4556948428ba4957093aea8ba" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 41603 } }
I20250813 20:00:03.961752 16939 leader_election.cc:304] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f59ecd4556948428ba4957093aea8ba; no voters:
I20250813 20:00:03.963347 16939 leader_election.cc:290] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 20:00:03.964079 16944 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [term 1 FOLLOWER]: Leader election won for term 1
I20250813 20:00:03.966619 16944 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [term 1 LEADER]: Becoming Leader. State: Replica: 9f59ecd4556948428ba4957093aea8ba, State: Running, Role: LEADER
I20250813 20:00:03.967326 16944 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59ecd4556948428ba4957093aea8ba" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 41603 } }
I20250813 20:00:03.967576 16939 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [sys.catalog]: configured and running, proceeding with master startup.
I20250813 20:00:03.976202 16945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f59ecd4556948428ba4957093aea8ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59ecd4556948428ba4957093aea8ba" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 41603 } } }
I20250813 20:00:03.977058 16945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [sys.catalog]: This master's current role is: LEADER
I20250813 20:00:03.977691 16946 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f59ecd4556948428ba4957093aea8ba. Latest consensus state: current_term: 1 leader_uuid: "9f59ecd4556948428ba4957093aea8ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f59ecd4556948428ba4957093aea8ba" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 41603 } } }
I20250813 20:00:03.978245 16946 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba [sys.catalog]: This master's current role is: LEADER
I20250813 20:00:03.981243 16953 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 20:00:03.991858 16953 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 20:00:03.994585 16961 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250813 20:00:04.007225 16953 catalog_manager.cc:1349] Generated new cluster ID: d020ee566d8d45c4b9a56b55746e6d21
I20250813 20:00:04.007599 16953 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 20:00:04.024741 16953 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 20:00:04.026202 16953 catalog_manager.cc:1506] Loading token signing keys...
I20250813 20:00:04.044644 16953 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9f59ecd4556948428ba4957093aea8ba: Generated new TSK 0
I20250813 20:00:04.045508 16953 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 20:00:04.070381 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:0
--local_ip_for_outbound_sockets=127.11.36.193
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:41603
--builtin_ntp_servers=127.11.36.212:40027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250813 20:00:04.352948 16963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:04.353422 16963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 20:00:04.353864 16963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:04.382858 16963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 20:00:04.383594 16963 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 20:00:04.417946 16963 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:40027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:41603
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 20:00:04.419178 16963 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:04.420696 16963 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:04.432252 16969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:04.433552 16970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:05.782344 16971 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1343 milliseconds
W20250813 20:00:05.785058 16972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:05.787094 16963 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.354s user 0.000s sys 0.008s
W20250813 20:00:05.787441 16963 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.354s user 0.000s sys 0.008s
I20250813 20:00:05.787710 16963 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 20:00:05.789189 16963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 20:00:05.792590 16963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 20:00:05.794086 16963 hybrid_clock.cc:648] HybridClock initialized: now 1755115205794014 us; error 74 us; skew 500 ppm
I20250813 20:00:05.795151 16963 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:05.803601 16963 webserver.cc:480] Webserver started at http://127.11.36.193:41841/ using document root <none> and password file <none>
I20250813 20:00:05.804600 16963 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:05.804852 16963 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:05.805318 16963 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 20:00:05.809527 16963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2a42e9d7fed74d1ba86b22b1f12c820c"
format_stamp: "Formatted at 2025-08-13 20:00:05 on dist-test-slave-l098"
I20250813 20:00:05.810555 16963 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2a42e9d7fed74d1ba86b22b1f12c820c"
format_stamp: "Formatted at 2025-08-13 20:00:05 on dist-test-slave-l098"
I20250813 20:00:05.817860 16963 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20250813 20:00:05.823896 16979 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:05.825058 16963 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250813 20:00:05.825374 16963 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2a42e9d7fed74d1ba86b22b1f12c820c"
format_stamp: "Formatted at 2025-08-13 20:00:05 on dist-test-slave-l098"
I20250813 20:00:05.825680 16963 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:05.892002 16963 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:05.893867 16963 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:05.894421 16963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:05.913987 16963 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 20:00:05.918603 16963 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 20:00:05.918828 16963 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:05.919065 16963 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 20:00:05.919214 16963 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:06.078190 16963 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:42507
I20250813 20:00:06.078352 17091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:42507 every 8 connection(s)
I20250813 20:00:06.080713 16963 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250813 20:00:06.082211 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 16963
I20250813 20:00:06.082722 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755115061406653-11411-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250813 20:00:06.106283 17092 heartbeater.cc:344] Connected to a master server at 127.11.36.254:41603
I20250813 20:00:06.106703 17092 heartbeater.cc:461] Registering TS with master...
I20250813 20:00:06.107635 17092 heartbeater.cc:507] Master 127.11.36.254:41603 requested a full tablet report, sending...
I20250813 20:00:06.109930 16904 ts_manager.cc:194] Registered new tserver with Master: 2a42e9d7fed74d1ba86b22b1f12c820c (127.11.36.193:42507)
I20250813 20:00:06.111690 16904 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:34041
I20250813 20:00:06.119984 11411 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250813 20:00:06.159601 16904 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57712:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250813 20:00:06.220232 17027 tablet_service.cc:1468] Processing CreateTablet for tablet d013ea1057bc4fab91d8f28ad822bd40 (DEFAULT_TABLE table=TestTable [id=3d01901900c0468781d6f5a08246d99c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 20:00:06.222045 17027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d013ea1057bc4fab91d8f28ad822bd40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 20:00:06.247570 17107 tablet_bootstrap.cc:492] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c: Bootstrap starting.
I20250813 20:00:06.253033 17107 tablet_bootstrap.cc:654] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c: Neither blocks nor log segments found. Creating new log.
I20250813 20:00:06.254739 17107 log.cc:826] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c: Log is configured to *not* fsync() on all Append() calls
I20250813 20:00:06.259694 17107 tablet_bootstrap.cc:492] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c: No bootstrap required, opened a new log
I20250813 20:00:06.260217 17107 ts_tablet_manager.cc:1397] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.007s
I20250813 20:00:06.276352 17107 raft_consensus.cc:357] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a42e9d7fed74d1ba86b22b1f12c820c" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42507 } }
I20250813 20:00:06.276940 17107 raft_consensus.cc:383] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 20:00:06.277186 17107 raft_consensus.cc:738] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a42e9d7fed74d1ba86b22b1f12c820c, State: Initialized, Role: FOLLOWER
I20250813 20:00:06.277915 17107 consensus_queue.cc:260] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a42e9d7fed74d1ba86b22b1f12c820c" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42507 } }
I20250813 20:00:06.278410 17107 raft_consensus.cc:397] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 20:00:06.278661 17107 raft_consensus.cc:491] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 20:00:06.278939 17107 raft_consensus.cc:3058] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [term 0 FOLLOWER]: Advancing to term 1
I20250813 20:00:06.282847 17107 raft_consensus.cc:513] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a42e9d7fed74d1ba86b22b1f12c820c" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42507 } }
I20250813 20:00:06.283579 17107 leader_election.cc:304] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2a42e9d7fed74d1ba86b22b1f12c820c; no voters:
I20250813 20:00:06.285332 17107 leader_election.cc:290] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 20:00:06.285835 17109 raft_consensus.cc:2802] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [term 1 FOLLOWER]: Leader election won for term 1
I20250813 20:00:06.289016 17092 heartbeater.cc:499] Master 127.11.36.254:41603 was elected leader, sending a full tablet report...
I20250813 20:00:06.289754 17107 ts_tablet_manager.cc:1428] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c: Time spent starting tablet: real 0.029s user 0.022s sys 0.006s
I20250813 20:00:06.290863 17109 raft_consensus.cc:695] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [term 1 LEADER]: Becoming Leader. State: Replica: 2a42e9d7fed74d1ba86b22b1f12c820c, State: Running, Role: LEADER
I20250813 20:00:06.291718 17109 consensus_queue.cc:237] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a42e9d7fed74d1ba86b22b1f12c820c" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42507 } }
I20250813 20:00:06.303004 16904 catalog_manager.cc:5582] T d013ea1057bc4fab91d8f28ad822bd40 P 2a42e9d7fed74d1ba86b22b1f12c820c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a42e9d7fed74d1ba86b22b1f12c820c (127.11.36.193). New cstate: current_term: 1 leader_uuid: "2a42e9d7fed74d1ba86b22b1f12c820c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2a42e9d7fed74d1ba86b22b1f12c820c" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 42507 } health_report { overall_health: HEALTHY } } }
I20250813 20:00:06.320894 11411 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250813 20:00:06.323994 11411 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2a42e9d7fed74d1ba86b22b1f12c820c to finish bootstrapping
I20250813 20:00:08.958207 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 16963
I20250813 20:00:08.981262 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 16870
2025-08-13T20:00:09Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7399 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7400 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250813 20:00:09.036923 11411 test_util.cc:276] Using random seed: -1701637024
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (11 ms)
[----------] 1 test from ListTableCliParamTest (11 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-13T20:00:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-13T20:00:09Z Disabled control of system clock
I20250813 20:00:09.085954 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:42849
--webserver_interface=127.11.36.254
--webserver_port=0
--builtin_ntp_servers=127.11.36.212:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:42849 with env {}
W20250813 20:00:09.369259 17135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:09.369884 17135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 20:00:09.370335 17135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:09.399760 17135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 20:00:09.400087 17135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 20:00:09.400333 17135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 20:00:09.400559 17135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 20:00:09.435712 17135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:42849
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:42849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 20:00:09.437147 17135 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:09.439124 17135 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:09.449457 17141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:10.853628 17140 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 17135
W20250813 20:00:11.134573 17140 kernel_stack_watchdog.cc:198] Thread 17135 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250813 20:00:09.450059 17142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:11.135823 17135 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.686s user 0.559s sys 1.127s
W20250813 20:00:11.136305 17135 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.687s user 0.559s sys 1.127s
W20250813 20:00:11.137876 17144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:11.139423 17143 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1685 milliseconds
I20250813 20:00:11.139462 17135 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 20:00:11.140730 17135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 20:00:11.143014 17135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 20:00:11.144320 17135 hybrid_clock.cc:648] HybridClock initialized: now 1755115211144284 us; error 45 us; skew 500 ppm
I20250813 20:00:11.145071 17135 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:11.150617 17135 webserver.cc:480] Webserver started at http://127.11.36.254:36853/ using document root <none> and password file <none>
I20250813 20:00:11.151471 17135 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:11.151647 17135 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:11.152047 17135 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 20:00:11.156147 17135 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/instance:
uuid: "6038857e2eb14e06aef65ca9c151b4f7"
format_stamp: "Formatted at 2025-08-13 20:00:11 on dist-test-slave-l098"
I20250813 20:00:11.157094 17135 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal/instance:
uuid: "6038857e2eb14e06aef65ca9c151b4f7"
format_stamp: "Formatted at 2025-08-13 20:00:11 on dist-test-slave-l098"
I20250813 20:00:11.163460 17135 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250813 20:00:11.168285 17151 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:11.169201 17135 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250813 20:00:11.169483 17135 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
uuid: "6038857e2eb14e06aef65ca9c151b4f7"
format_stamp: "Formatted at 2025-08-13 20:00:11 on dist-test-slave-l098"
I20250813 20:00:11.169811 17135 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:11.213896 17135 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:11.215210 17135 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:11.215607 17135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:11.279151 17135 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:42849
I20250813 20:00:11.279208 17202 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:42849 every 8 connection(s)
I20250813 20:00:11.281652 17135 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/info.pb
I20250813 20:00:11.286300 17203 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 20:00:11.288502 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 17135
I20250813 20:00:11.288933 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal/instance
I20250813 20:00:11.305871 17203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7: Bootstrap starting.
I20250813 20:00:11.311061 17203 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7: Neither blocks nor log segments found. Creating new log.
I20250813 20:00:11.312927 17203 log.cc:826] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7: Log is configured to *not* fsync() on all Append() calls
I20250813 20:00:11.317447 17203 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7: No bootstrap required, opened a new log
I20250813 20:00:11.332988 17203 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038857e2eb14e06aef65ca9c151b4f7" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } }
I20250813 20:00:11.333573 17203 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 20:00:11.333747 17203 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6038857e2eb14e06aef65ca9c151b4f7, State: Initialized, Role: FOLLOWER
I20250813 20:00:11.334324 17203 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038857e2eb14e06aef65ca9c151b4f7" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } }
I20250813 20:00:11.334748 17203 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 20:00:11.334964 17203 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 20:00:11.335253 17203 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [term 0 FOLLOWER]: Advancing to term 1
I20250813 20:00:11.339099 17203 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038857e2eb14e06aef65ca9c151b4f7" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } }
I20250813 20:00:11.339708 17203 leader_election.cc:304] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6038857e2eb14e06aef65ca9c151b4f7; no voters:
I20250813 20:00:11.341327 17203 leader_election.cc:290] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 20:00:11.342099 17208 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 20:00:11.344336 17208 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [term 1 LEADER]: Becoming Leader. State: Replica: 6038857e2eb14e06aef65ca9c151b4f7, State: Running, Role: LEADER
I20250813 20:00:11.345008 17208 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038857e2eb14e06aef65ca9c151b4f7" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } }
I20250813 20:00:11.345929 17203 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 20:00:11.354372 17209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6038857e2eb14e06aef65ca9c151b4f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038857e2eb14e06aef65ca9c151b4f7" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } } }
I20250813 20:00:11.354395 17210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6038857e2eb14e06aef65ca9c151b4f7. Latest consensus state: current_term: 1 leader_uuid: "6038857e2eb14e06aef65ca9c151b4f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6038857e2eb14e06aef65ca9c151b4f7" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } } }
I20250813 20:00:11.355146 17210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [sys.catalog]: This master's current role is: LEADER
I20250813 20:00:11.355146 17209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7 [sys.catalog]: This master's current role is: LEADER
I20250813 20:00:11.358801 17216 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 20:00:11.369242 17216 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 20:00:11.385439 17216 catalog_manager.cc:1349] Generated new cluster ID: 68017f3b27184f0db7b54f76aa4b436a
I20250813 20:00:11.385748 17216 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 20:00:11.408005 17216 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 20:00:11.409399 17216 catalog_manager.cc:1506] Loading token signing keys...
I20250813 20:00:11.420156 17216 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6038857e2eb14e06aef65ca9c151b4f7: Generated new TSK 0
I20250813 20:00:11.421001 17216 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250813 20:00:11.445933 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:0
--local_ip_for_outbound_sockets=127.11.36.193
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:42849
--builtin_ntp_servers=127.11.36.212:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250813 20:00:11.754657 17227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:11.755148 17227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 20:00:11.755566 17227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:11.783694 17227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 20:00:11.784544 17227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 20:00:11.816268 17227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=0
--tserver_master_addrs=127.11.36.254:42849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 20:00:11.817466 17227 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:11.818918 17227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:11.830225 17233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:11.831298 17234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:13.055086 17236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:13.057600 17227 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.227s user 0.463s sys 0.756s
W20250813 20:00:13.057942 17227 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.227s user 0.463s sys 0.756s
I20250813 20:00:13.058221 17227 server_base.cc:1047] running on GCE node
I20250813 20:00:13.059590 17227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 20:00:13.062505 17227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 20:00:13.064052 17227 hybrid_clock.cc:648] HybridClock initialized: now 1755115213063954 us; error 97 us; skew 500 ppm
I20250813 20:00:13.065192 17227 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:13.073876 17227 webserver.cc:480] Webserver started at http://127.11.36.193:43613/ using document root <none> and password file <none>
I20250813 20:00:13.075256 17227 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:13.075541 17227 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:13.076174 17227 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 20:00:13.083019 17227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/instance:
uuid: "c6c0c58e02e84a97ae87212df4e00ac0"
format_stamp: "Formatted at 2025-08-13 20:00:13 on dist-test-slave-l098"
I20250813 20:00:13.084481 17227 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal/instance:
uuid: "c6c0c58e02e84a97ae87212df4e00ac0"
format_stamp: "Formatted at 2025-08-13 20:00:13 on dist-test-slave-l098"
I20250813 20:00:13.094180 17227 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.000s sys 0.008s
I20250813 20:00:13.102136 17244 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:13.103452 17227 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20250813 20:00:13.103834 17227 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal
uuid: "c6c0c58e02e84a97ae87212df4e00ac0"
format_stamp: "Formatted at 2025-08-13 20:00:13 on dist-test-slave-l098"
I20250813 20:00:13.104283 17227 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:13.197506 17227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:13.198961 17227 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:13.199368 17227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:13.202132 17227 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 20:00:13.206571 17227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 20:00:13.206789 17227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:13.207001 17227 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 20:00:13.207141 17227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:13.361485 17227 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:37389
I20250813 20:00:13.361644 17356 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:37389 every 8 connection(s)
I20250813 20:00:13.364030 17227 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/info.pb
I20250813 20:00:13.367906 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 17227
I20250813 20:00:13.368626 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal/instance
I20250813 20:00:13.376820 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.194:0
--local_ip_for_outbound_sockets=127.11.36.194
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:42849
--builtin_ntp_servers=127.11.36.212:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250813 20:00:13.397595 17357 heartbeater.cc:344] Connected to a master server at 127.11.36.254:42849
I20250813 20:00:13.398002 17357 heartbeater.cc:461] Registering TS with master...
I20250813 20:00:13.398919 17357 heartbeater.cc:507] Master 127.11.36.254:42849 requested a full tablet report, sending...
I20250813 20:00:13.401404 17168 ts_manager.cc:194] Registered new tserver with Master: c6c0c58e02e84a97ae87212df4e00ac0 (127.11.36.193:37389)
I20250813 20:00:13.403287 17168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:51955
W20250813 20:00:13.707988 17361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:13.708496 17361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 20:00:13.709009 17361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:13.737171 17361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 20:00:13.738003 17361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.194
I20250813 20:00:13.770279 17361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.36.194
--webserver_port=0
--tserver_master_addrs=127.11.36.254:42849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.194
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 20:00:13.771633 17361 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:13.773139 17361 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:13.784745 17367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 20:00:14.406419 17357 heartbeater.cc:499] Master 127.11.36.254:42849 was elected leader, sending a full tablet report...
W20250813 20:00:13.784989 17368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:15.011518 17370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:15.013406 17369 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1223 milliseconds
W20250813 20:00:15.014448 17361 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.230s user 0.375s sys 0.837s
W20250813 20:00:15.014683 17361 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.230s user 0.375s sys 0.837s
I20250813 20:00:15.014882 17361 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 20:00:15.015873 17361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 20:00:15.018090 17361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 20:00:15.019445 17361 hybrid_clock.cc:648] HybridClock initialized: now 1755115215019420 us; error 38 us; skew 500 ppm
I20250813 20:00:15.020208 17361 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:15.027278 17361 webserver.cc:480] Webserver started at http://127.11.36.194:33341/ using document root <none> and password file <none>
I20250813 20:00:15.028218 17361 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:15.028411 17361 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:15.028831 17361 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 20:00:15.032860 17361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/instance:
uuid: "ba454781df5f46cd99240cf8036588a1"
format_stamp: "Formatted at 2025-08-13 20:00:15 on dist-test-slave-l098"
I20250813 20:00:15.033951 17361 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal/instance:
uuid: "ba454781df5f46cd99240cf8036588a1"
format_stamp: "Formatted at 2025-08-13 20:00:15 on dist-test-slave-l098"
I20250813 20:00:15.041090 17361 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250813 20:00:15.046934 17377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:15.047962 17361 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250813 20:00:15.048295 17361 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal
uuid: "ba454781df5f46cd99240cf8036588a1"
format_stamp: "Formatted at 2025-08-13 20:00:15 on dist-test-slave-l098"
I20250813 20:00:15.048626 17361 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:15.118531 17361 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:15.119892 17361 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:15.120314 17361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:15.122702 17361 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 20:00:15.126554 17361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 20:00:15.126735 17361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:15.126971 17361 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 20:00:15.127110 17361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:15.253846 17361 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.194:39745
I20250813 20:00:15.253921 17489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.194:39745 every 8 connection(s)
I20250813 20:00:15.256462 17361 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/info.pb
I20250813 20:00:15.264966 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 17361
I20250813 20:00:15.265355 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal/instance
I20250813 20:00:15.271216 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.195:0
--local_ip_for_outbound_sockets=127.11.36.195
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:42849
--builtin_ntp_servers=127.11.36.212:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250813 20:00:15.276027 17490 heartbeater.cc:344] Connected to a master server at 127.11.36.254:42849
I20250813 20:00:15.276547 17490 heartbeater.cc:461] Registering TS with master...
I20250813 20:00:15.277776 17490 heartbeater.cc:507] Master 127.11.36.254:42849 requested a full tablet report, sending...
I20250813 20:00:15.280192 17168 ts_manager.cc:194] Registered new tserver with Master: ba454781df5f46cd99240cf8036588a1 (127.11.36.194:39745)
I20250813 20:00:15.281339 17168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.194:34189
W20250813 20:00:15.569106 17494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:15.569531 17494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 20:00:15.569918 17494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:15.598971 17494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 20:00:15.599694 17494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.195
I20250813 20:00:15.631703 17494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.36.195
--webserver_port=0
--tserver_master_addrs=127.11.36.254:42849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.195
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 20:00:15.633008 17494 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:15.634445 17494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:15.645628 17500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 20:00:16.284328 17490 heartbeater.cc:499] Master 127.11.36.254:42849 was elected leader, sending a full tablet report...
W20250813 20:00:15.646226 17501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:16.782310 17503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:16.783821 17502 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1134 milliseconds
I20250813 20:00:16.783968 17494 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 20:00:16.785115 17494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 20:00:16.788724 17494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 20:00:16.790145 17494 hybrid_clock.cc:648] HybridClock initialized: now 1755115216790110 us; error 40 us; skew 500 ppm
I20250813 20:00:16.790892 17494 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:16.797153 17494 webserver.cc:480] Webserver started at http://127.11.36.195:39145/ using document root <none> and password file <none>
I20250813 20:00:16.798053 17494 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:16.798249 17494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:16.798722 17494 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 20:00:16.803189 17494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/instance:
uuid: "b25247ef9cff4bb48e98ed5ac8134eb3"
format_stamp: "Formatted at 2025-08-13 20:00:16 on dist-test-slave-l098"
I20250813 20:00:16.804247 17494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal/instance:
uuid: "b25247ef9cff4bb48e98ed5ac8134eb3"
format_stamp: "Formatted at 2025-08-13 20:00:16 on dist-test-slave-l098"
I20250813 20:00:16.810847 17494 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250813 20:00:16.816176 17510 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:16.817075 17494 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250813 20:00:16.817346 17494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal
uuid: "b25247ef9cff4bb48e98ed5ac8134eb3"
format_stamp: "Formatted at 2025-08-13 20:00:16 on dist-test-slave-l098"
I20250813 20:00:16.817664 17494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:16.867390 17494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:16.868808 17494 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:16.869220 17494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:16.871511 17494 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 20:00:16.875483 17494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250813 20:00:16.875686 17494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:16.875923 17494 ts_tablet_manager.cc:610] Registered 0 tablets
I20250813 20:00:16.876097 17494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20250813 20:00:17.005609 17494 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.195:36881
I20250813 20:00:17.005668 17622 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.195:36881 every 8 connection(s)
I20250813 20:00:17.008235 17494 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/info.pb
I20250813 20:00:17.016567 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 17494
I20250813 20:00:17.017125 11411 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal/instance
I20250813 20:00:17.039068 17623 heartbeater.cc:344] Connected to a master server at 127.11.36.254:42849
I20250813 20:00:17.039471 17623 heartbeater.cc:461] Registering TS with master...
I20250813 20:00:17.040493 17623 heartbeater.cc:507] Master 127.11.36.254:42849 requested a full tablet report, sending...
I20250813 20:00:17.042423 17168 ts_manager.cc:194] Registered new tserver with Master: b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881)
I20250813 20:00:17.043555 17168 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.195:47757
I20250813 20:00:17.050187 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 20:00:17.077939 11411 test_util.cc:276] Using random seed: -1693596002
I20250813 20:00:17.130322 17168 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45056:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250813 20:00:17.132771 17168 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 20:00:17.181224 17292 tablet_service.cc:1468] Processing CreateTablet for tablet 48d3298b28c1457f9aa9b7553cfcd0a4 (DEFAULT_TABLE table=pre_rebuild [id=3531105ac495418daf6093b4f40c4144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 20:00:17.183081 17292 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48d3298b28c1457f9aa9b7553cfcd0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 20:00:17.186323 17425 tablet_service.cc:1468] Processing CreateTablet for tablet 48d3298b28c1457f9aa9b7553cfcd0a4 (DEFAULT_TABLE table=pre_rebuild [id=3531105ac495418daf6093b4f40c4144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 20:00:17.188218 17425 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48d3298b28c1457f9aa9b7553cfcd0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 20:00:17.193806 17558 tablet_service.cc:1468] Processing CreateTablet for tablet 48d3298b28c1457f9aa9b7553cfcd0a4 (DEFAULT_TABLE table=pre_rebuild [id=3531105ac495418daf6093b4f40c4144]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 20:00:17.195674 17558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48d3298b28c1457f9aa9b7553cfcd0a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 20:00:17.204567 17647 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Bootstrap starting.
I20250813 20:00:17.210116 17647 tablet_bootstrap.cc:654] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Neither blocks nor log segments found. Creating new log.
I20250813 20:00:17.211803 17647 log.cc:826] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Log is configured to *not* fsync() on all Append() calls
I20250813 20:00:17.214520 17648 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Bootstrap starting.
I20250813 20:00:17.218947 17647 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: No bootstrap required, opened a new log
I20250813 20:00:17.219584 17647 ts_tablet_manager.cc:1397] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.014s
I20250813 20:00:17.222048 17648 tablet_bootstrap.cc:654] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Neither blocks nor log segments found. Creating new log.
I20250813 20:00:17.224025 17650 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Bootstrap starting.
I20250813 20:00:17.224442 17648 log.cc:826] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Log is configured to *not* fsync() on all Append() calls
I20250813 20:00:17.230192 17648 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: No bootstrap required, opened a new log
I20250813 20:00:17.230288 17650 tablet_bootstrap.cc:654] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Neither blocks nor log segments found. Creating new log.
I20250813 20:00:17.230695 17648 ts_tablet_manager.cc:1397] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.005s
I20250813 20:00:17.232663 17650 log.cc:826] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Log is configured to *not* fsync() on all Append() calls
I20250813 20:00:17.237314 17650 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: No bootstrap required, opened a new log
I20250813 20:00:17.237696 17650 ts_tablet_manager.cc:1397] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.011s
I20250813 20:00:17.245162 17647 raft_consensus.cc:357] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:17.246111 17647 raft_consensus.cc:383] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 20:00:17.246397 17647 raft_consensus.cc:738] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c0c58e02e84a97ae87212df4e00ac0, State: Initialized, Role: FOLLOWER
I20250813 20:00:17.247298 17647 consensus_queue.cc:260] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:17.257264 17650 raft_consensus.cc:357] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:17.258262 17650 raft_consensus.cc:383] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 20:00:17.258566 17650 raft_consensus.cc:738] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25247ef9cff4bb48e98ed5ac8134eb3, State: Initialized, Role: FOLLOWER
I20250813 20:00:17.259608 17650 consensus_queue.cc:260] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:17.259450 17648 raft_consensus.cc:357] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:17.260689 17648 raft_consensus.cc:383] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 20:00:17.261022 17648 raft_consensus.cc:738] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba454781df5f46cd99240cf8036588a1, State: Initialized, Role: FOLLOWER
I20250813 20:00:17.262068 17648 consensus_queue.cc:260] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:17.277491 17623 heartbeater.cc:499] Master 127.11.36.254:42849 was elected leader, sending a full tablet report...
I20250813 20:00:17.279546 17650 ts_tablet_manager.cc:1428] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Time spent starting tablet: real 0.042s user 0.028s sys 0.000s
I20250813 20:00:17.276954 17647 ts_tablet_manager.cc:1428] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Time spent starting tablet: real 0.057s user 0.034s sys 0.009s
I20250813 20:00:17.287083 17648 ts_tablet_manager.cc:1428] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Time spent starting tablet: real 0.056s user 0.035s sys 0.019s
I20250813 20:00:17.359081 17654 raft_consensus.cc:491] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 20:00:17.359714 17654 raft_consensus.cc:513] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:17.363006 17654 leader_election.cc:290] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6c0c58e02e84a97ae87212df4e00ac0 (127.11.36.193:37389), ba454781df5f46cd99240cf8036588a1 (127.11.36.194:39745)
W20250813 20:00:17.372364 17358 tablet.cc:2378] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 20:00:17.374486 17312 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48d3298b28c1457f9aa9b7553cfcd0a4" candidate_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" is_pre_election: true
I20250813 20:00:17.374593 17445 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48d3298b28c1457f9aa9b7553cfcd0a4" candidate_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba454781df5f46cd99240cf8036588a1" is_pre_election: true
I20250813 20:00:17.375303 17312 raft_consensus.cc:2466] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b25247ef9cff4bb48e98ed5ac8134eb3 in term 0.
I20250813 20:00:17.375432 17445 raft_consensus.cc:2466] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b25247ef9cff4bb48e98ed5ac8134eb3 in term 0.
I20250813 20:00:17.376714 17513 leader_election.cc:304] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b25247ef9cff4bb48e98ed5ac8134eb3, c6c0c58e02e84a97ae87212df4e00ac0; no voters:
I20250813 20:00:17.377524 17654 raft_consensus.cc:2802] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 20:00:17.377794 17654 raft_consensus.cc:491] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 20:00:17.378028 17654 raft_consensus.cc:3058] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Advancing to term 1
I20250813 20:00:17.382373 17654 raft_consensus.cc:513] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:17.383744 17654 leader_election.cc:290] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [CANDIDATE]: Term 1 election: Requested vote from peers c6c0c58e02e84a97ae87212df4e00ac0 (127.11.36.193:37389), ba454781df5f46cd99240cf8036588a1 (127.11.36.194:39745)
I20250813 20:00:17.384707 17445 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48d3298b28c1457f9aa9b7553cfcd0a4" candidate_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba454781df5f46cd99240cf8036588a1"
I20250813 20:00:17.384771 17312 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48d3298b28c1457f9aa9b7553cfcd0a4" candidate_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c0c58e02e84a97ae87212df4e00ac0"
I20250813 20:00:17.385124 17445 raft_consensus.cc:3058] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Advancing to term 1
I20250813 20:00:17.385268 17312 raft_consensus.cc:3058] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 0 FOLLOWER]: Advancing to term 1
I20250813 20:00:17.389412 17445 raft_consensus.cc:2466] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b25247ef9cff4bb48e98ed5ac8134eb3 in term 1.
I20250813 20:00:17.390305 17514 leader_election.cc:304] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b25247ef9cff4bb48e98ed5ac8134eb3, ba454781df5f46cd99240cf8036588a1; no voters:
I20250813 20:00:17.390913 17654 raft_consensus.cc:2802] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 20:00:17.391587 17312 raft_consensus.cc:2466] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b25247ef9cff4bb48e98ed5ac8134eb3 in term 1.
I20250813 20:00:17.392889 17654 raft_consensus.cc:695] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 1 LEADER]: Becoming Leader. State: Replica: b25247ef9cff4bb48e98ed5ac8134eb3, State: Running, Role: LEADER
I20250813 20:00:17.393694 17654 consensus_queue.cc:237] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:17.404340 17166 catalog_manager.cc:5582] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 reported cstate change: term changed from 0 to 1, leader changed from <none> to b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195). New cstate: current_term: 1 leader_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } health_report { overall_health: UNKNOWN } } }
W20250813 20:00:17.512962 17624 tablet.cc:2378] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 20:00:17.513070 17491 tablet.cc:2378] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 20:00:17.537809 17312 raft_consensus.cc:1273] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Refusing update from remote peer b25247ef9cff4bb48e98ed5ac8134eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250813 20:00:17.537897 17445 raft_consensus.cc:1273] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 1 FOLLOWER]: Refusing update from remote peer b25247ef9cff4bb48e98ed5ac8134eb3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250813 20:00:17.539220 17654 consensus_queue.cc:1035] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250813 20:00:17.540227 17657 consensus_queue.cc:1035] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 20:00:17.564363 17664 mvcc.cc:204] Tried to move back new op lower bound from 7188951931019329536 to 7188951930451533824. Current Snapshot: MvccSnapshot[applied={T|T < 7188951931019329536}]
I20250813 20:00:17.569015 17666 mvcc.cc:204] Tried to move back new op lower bound from 7188951931019329536 to 7188951930451533824. Current Snapshot: MvccSnapshot[applied={T|T < 7188951931019329536}]
I20250813 20:00:17.580056 17665 mvcc.cc:204] Tried to move back new op lower bound from 7188951931019329536 to 7188951930451533824. Current Snapshot: MvccSnapshot[applied={T|T < 7188951931019329536}]
I20250813 20:00:22.255184 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 17135
W20250813 20:00:22.584837 17694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:22.585428 17694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:22.614375 17694 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250813 20:00:22.629911 17490 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:42849 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:42849: connect: Connection refused (error 111)
W20250813 20:00:22.638324 17357 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:42849 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:42849: connect: Connection refused (error 111)
W20250813 20:00:22.639871 17623 heartbeater.cc:646] Failed to heartbeat to 127.11.36.254:42849 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.11.36.254:42849: connect: Connection refused (error 111)
W20250813 20:00:24.013907 17694 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.352s user 0.500s sys 0.836s
W20250813 20:00:24.014344 17694 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.353s user 0.501s sys 0.837s
I20250813 20:00:24.141885 17694 minidump.cc:252] Setting minidump size limit to 20M
I20250813 20:00:24.144354 17694 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:24.145373 17694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:24.157274 17731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:24.157356 17732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:24.319371 17734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 20:00:24.319880 17694 server_base.cc:1047] running on GCE node
I20250813 20:00:24.321336 17694 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250813 20:00:24.321964 17694 hybrid_clock.cc:648] HybridClock initialized: now 1755115224321933 us; error 50009 us; skew 500 ppm
I20250813 20:00:24.322907 17694 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:24.328225 17694 webserver.cc:480] Webserver started at http://0.0.0.0:46621/ using document root <none> and password file <none>
I20250813 20:00:24.329378 17694 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:24.329681 17694 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:24.330209 17694 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250813 20:00:24.337280 17694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/instance:
uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1"
format_stamp: "Formatted at 2025-08-13 20:00:24 on dist-test-slave-l098"
I20250813 20:00:24.338393 17694 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal/instance:
uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1"
format_stamp: "Formatted at 2025-08-13 20:00:24 on dist-test-slave-l098"
I20250813 20:00:24.344451 17694 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20250813 20:00:24.348856 17739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:24.349727 17694 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250813 20:00:24.350016 17694 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1"
format_stamp: "Formatted at 2025-08-13 20:00:24 on dist-test-slave-l098"
I20250813 20:00:24.350390 17694 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:24.385308 17694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:24.387207 17694 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:24.387737 17694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:24.393460 17694 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 20:00:24.412262 17694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: Bootstrap starting.
I20250813 20:00:24.416916 17694 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: Neither blocks nor log segments found. Creating new log.
I20250813 20:00:24.418532 17694 log.cc:826] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: Log is configured to *not* fsync() on all Append() calls
I20250813 20:00:24.422484 17694 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: No bootstrap required, opened a new log
I20250813 20:00:24.437196 17694 raft_consensus.cc:357] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER }
I20250813 20:00:24.437680 17694 raft_consensus.cc:383] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 20:00:24.437922 17694 raft_consensus.cc:738] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed87e9d4a7084a3aa2c026ae1c329ea1, State: Initialized, Role: FOLLOWER
I20250813 20:00:24.438544 17694 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER }
I20250813 20:00:24.438999 17694 raft_consensus.cc:397] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 20:00:24.439239 17694 raft_consensus.cc:491] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 20:00:24.439523 17694 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 0 FOLLOWER]: Advancing to term 1
I20250813 20:00:24.443156 17694 raft_consensus.cc:513] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER }
I20250813 20:00:24.443781 17694 leader_election.cc:304] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed87e9d4a7084a3aa2c026ae1c329ea1; no voters:
I20250813 20:00:24.445447 17694 leader_election.cc:290] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250813 20:00:24.445724 17746 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 20:00:24.448943 17746 raft_consensus.cc:695] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 1 LEADER]: Becoming Leader. State: Replica: ed87e9d4a7084a3aa2c026ae1c329ea1, State: Running, Role: LEADER
I20250813 20:00:24.449633 17746 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER }
I20250813 20:00:24.461668 17747 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER } }
I20250813 20:00:24.461890 17748 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed87e9d4a7084a3aa2c026ae1c329ea1. Latest consensus state: current_term: 1 leader_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER } }
I20250813 20:00:24.462515 17748 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [sys.catalog]: This master's current role is: LEADER
I20250813 20:00:24.462550 17747 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [sys.catalog]: This master's current role is: LEADER
I20250813 20:00:24.468726 17694 tablet_replica.cc:331] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: stopping tablet replica
I20250813 20:00:24.469344 17694 raft_consensus.cc:2241] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 1 LEADER]: Raft consensus shutting down.
I20250813 20:00:24.469808 17694 raft_consensus.cc:2270] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250813 20:00:24.472211 17694 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250813 20:00:24.472795 17694 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250813 20:00:24.500558 17694 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250813 20:00:25.530383 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 17227
I20250813 20:00:25.565974 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 17361
I20250813 20:00:25.599589 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 17494
I20250813 20:00:25.637859 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:42849
--webserver_interface=127.11.36.254
--webserver_port=36853
--builtin_ntp_servers=127.11.36.212:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.11.36.254:42849 with env {}
W20250813 20:00:25.921845 17756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:25.922418 17756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 20:00:25.922863 17756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:25.951542 17756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250813 20:00:25.951853 17756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 20:00:25.952095 17756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250813 20:00:25.952332 17756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250813 20:00:25.984130 17756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.11.36.254:42849
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.11.36.254:42849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.11.36.254
--webserver_port=36853
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 20:00:25.985361 17756 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:25.986869 17756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:25.996924 17762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:26.005667 17765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:25.997424 17763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:27.105806 17764 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1104 milliseconds
I20250813 20:00:27.105931 17756 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 20:00:27.107062 17756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 20:00:27.109468 17756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 20:00:27.110800 17756 hybrid_clock.cc:648] HybridClock initialized: now 1755115227110778 us; error 43 us; skew 500 ppm
I20250813 20:00:27.111485 17756 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:27.122031 17756 webserver.cc:480] Webserver started at http://127.11.36.254:36853/ using document root <none> and password file <none>
I20250813 20:00:27.122918 17756 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:27.123147 17756 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:27.131295 17756 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250813 20:00:27.137423 17772 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:27.138695 17756 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250813 20:00:27.139132 17756 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1"
format_stamp: "Formatted at 2025-08-13 20:00:24 on dist-test-slave-l098"
I20250813 20:00:27.142040 17756 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:27.197849 17756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:27.199204 17756 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:27.199627 17756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:27.266954 17756 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.254:42849
I20250813 20:00:27.266999 17823 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.254:42849 every 8 connection(s)
I20250813 20:00:27.269636 17756 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/info.pb
I20250813 20:00:27.276963 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 17756
I20250813 20:00:27.278965 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.193:37389
--local_ip_for_outbound_sockets=127.11.36.193
--tserver_master_addrs=127.11.36.254:42849
--webserver_port=43613
--webserver_interface=127.11.36.193
--builtin_ntp_servers=127.11.36.212:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250813 20:00:27.279922 17824 sys_catalog.cc:263] Verifying existing consensus state
I20250813 20:00:27.294998 17824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: Bootstrap starting.
I20250813 20:00:27.303961 17824 log.cc:826] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: Log is configured to *not* fsync() on all Append() calls
I20250813 20:00:27.314597 17824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 20:00:27.315244 17824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: Bootstrap complete.
I20250813 20:00:27.334744 17824 raft_consensus.cc:357] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } }
I20250813 20:00:27.335340 17824 raft_consensus.cc:738] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed87e9d4a7084a3aa2c026ae1c329ea1, State: Initialized, Role: FOLLOWER
I20250813 20:00:27.336054 17824 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } }
I20250813 20:00:27.336544 17824 raft_consensus.cc:397] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250813 20:00:27.336798 17824 raft_consensus.cc:491] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250813 20:00:27.337097 17824 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 1 FOLLOWER]: Advancing to term 2
I20250813 20:00:27.340896 17824 raft_consensus.cc:513] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } }
I20250813 20:00:27.341493 17824 leader_election.cc:304] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ed87e9d4a7084a3aa2c026ae1c329ea1; no voters:
I20250813 20:00:27.343170 17824 leader_election.cc:290] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250813 20:00:27.343780 17828 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 2 FOLLOWER]: Leader election won for term 2
I20250813 20:00:27.346537 17828 raft_consensus.cc:695] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [term 2 LEADER]: Becoming Leader. State: Replica: ed87e9d4a7084a3aa2c026ae1c329ea1, State: Running, Role: LEADER
I20250813 20:00:27.347350 17828 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } }
I20250813 20:00:27.347703 17824 sys_catalog.cc:564] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [sys.catalog]: configured and running, proceeding with master startup.
I20250813 20:00:27.358743 17829 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } } }
I20250813 20:00:27.359747 17829 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [sys.catalog]: This master's current role is: LEADER
I20250813 20:00:27.362313 17830 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed87e9d4a7084a3aa2c026ae1c329ea1. Latest consensus state: current_term: 2 leader_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ed87e9d4a7084a3aa2c026ae1c329ea1" member_type: VOTER last_known_addr { host: "127.11.36.254" port: 42849 } } }
I20250813 20:00:27.362978 17830 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1 [sys.catalog]: This master's current role is: LEADER
I20250813 20:00:27.365823 17834 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250813 20:00:27.380963 17834 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=627bdc5918184b8a826309aab13ed7a8]
I20250813 20:00:27.389108 17834 tablet_loader.cc:96] loaded metadata for tablet 48d3298b28c1457f9aa9b7553cfcd0a4 (table pre_rebuild [id=627bdc5918184b8a826309aab13ed7a8])
I20250813 20:00:27.391255 17834 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250813 20:00:27.419816 17834 catalog_manager.cc:1349] Generated new cluster ID: 16c1432df40d45e2ae6bd3472f1c1ead
I20250813 20:00:27.420239 17834 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250813 20:00:27.456317 17834 catalog_manager.cc:1372] Generated new certificate authority record
I20250813 20:00:27.458333 17834 catalog_manager.cc:1506] Loading token signing keys...
I20250813 20:00:27.474460 17834 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: Generated new TSK 0
I20250813 20:00:27.475606 17834 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250813 20:00:27.641585 17826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:27.642071 17826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 20:00:27.642527 17826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:27.671862 17826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 20:00:27.672741 17826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.193
I20250813 20:00:27.704849 17826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.193:37389
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.11.36.193
--webserver_port=43613
--tserver_master_addrs=127.11.36.254:42849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.193
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 20:00:27.706159 17826 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:27.707768 17826 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:27.719466 17852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:28.851750 17854 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250813 20:00:27.722070 17853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:28.854099 17855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:28.857717 17826 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.138s user 0.003s sys 0.004s
W20250813 20:00:28.858068 17826 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.139s user 0.003s sys 0.004s
I20250813 20:00:28.858309 17826 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 20:00:28.861070 17826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 20:00:28.864001 17826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 20:00:28.865507 17826 hybrid_clock.cc:648] HybridClock initialized: now 1755115228865451 us; error 78 us; skew 500 ppm
I20250813 20:00:28.866299 17826 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:28.876869 17826 webserver.cc:480] Webserver started at http://127.11.36.193:43613/ using document root <none> and password file <none>
I20250813 20:00:28.877861 17826 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:28.878228 17826 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:28.886809 17826 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20250813 20:00:28.892275 17862 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:28.893617 17826 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250813 20:00:28.893966 17826 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal
uuid: "c6c0c58e02e84a97ae87212df4e00ac0"
format_stamp: "Formatted at 2025-08-13 20:00:13 on dist-test-slave-l098"
I20250813 20:00:28.895968 17826 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:28.967820 17826 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:28.969273 17826 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:28.969738 17826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:28.972817 17826 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 20:00:28.979233 17869 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 20:00:28.987178 17826 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 20:00:28.987398 17826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.001s sys 0.002s
I20250813 20:00:28.987663 17826 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 20:00:28.992198 17826 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 20:00:28.992446 17826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250813 20:00:28.992838 17869 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Bootstrap starting.
I20250813 20:00:29.155839 17826 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.193:37389
I20250813 20:00:29.156009 17975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.193:37389 every 8 connection(s)
I20250813 20:00:29.159433 17826 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/info.pb
I20250813 20:00:29.164569 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 17826
I20250813 20:00:29.166488 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.194:39745
--local_ip_for_outbound_sockets=127.11.36.194
--tserver_master_addrs=127.11.36.254:42849
--webserver_port=33341
--webserver_interface=127.11.36.194
--builtin_ntp_servers=127.11.36.212:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250813 20:00:29.217854 17976 heartbeater.cc:344] Connected to a master server at 127.11.36.254:42849
I20250813 20:00:29.218307 17976 heartbeater.cc:461] Registering TS with master...
I20250813 20:00:29.219390 17976 heartbeater.cc:507] Master 127.11.36.254:42849 requested a full tablet report, sending...
I20250813 20:00:29.224000 17789 ts_manager.cc:194] Registered new tserver with Master: c6c0c58e02e84a97ae87212df4e00ac0 (127.11.36.193:37389)
I20250813 20:00:29.231333 17789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.193:50733
I20250813 20:00:29.311240 17869 log.cc:826] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Log is configured to *not* fsync() on all Append() calls
W20250813 20:00:29.614006 17980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:29.614670 17980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 20:00:29.615374 17980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:29.667997 17980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 20:00:29.669385 17980 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.194
I20250813 20:00:29.728587 17980 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.194:39745
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.11.36.194
--webserver_port=33341
--tserver_master_addrs=127.11.36.254:42849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.194
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 20:00:29.730161 17980 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:29.732168 17980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:29.746086 17987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 20:00:30.235555 17976 heartbeater.cc:499] Master 127.11.36.254:42849 was elected leader, sending a full tablet report...
W20250813 20:00:31.147696 17986 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 17980
W20250813 20:00:31.789206 17986 kernel_stack_watchdog.cc:198] Thread 17980 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 378ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250813 20:00:31.790983 17980 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 2.024s user 0.002s sys 0.000s
W20250813 20:00:31.791332 17980 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 2.024s user 0.002s sys 0.001s
W20250813 20:00:29.748975 17988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:31.793597 17992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:31.796525 17989 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 2043 milliseconds
I20250813 20:00:31.796530 17980 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250813 20:00:31.797660 17980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 20:00:31.800026 17980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 20:00:31.801457 17980 hybrid_clock.cc:648] HybridClock initialized: now 1755115231801415 us; error 52 us; skew 500 ppm
I20250813 20:00:31.802198 17980 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:31.808333 17980 webserver.cc:480] Webserver started at http://127.11.36.194:33341/ using document root <none> and password file <none>
I20250813 20:00:31.809231 17980 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:31.809444 17980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:31.817209 17980 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250813 20:00:31.821451 17997 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:31.822494 17980 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250813 20:00:31.822791 17980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal
uuid: "ba454781df5f46cd99240cf8036588a1"
format_stamp: "Formatted at 2025-08-13 20:00:15 on dist-test-slave-l098"
I20250813 20:00:31.824760 17980 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:31.909873 17980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:31.911264 17980 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:31.911690 17980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:31.914628 17980 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 20:00:31.920150 18004 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 20:00:31.930526 17980 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 20:00:31.930763 17980 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.002s sys 0.000s
I20250813 20:00:31.931005 17980 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 20:00:31.935376 17980 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 20:00:31.935588 17980 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.002s
I20250813 20:00:31.936025 18004 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Bootstrap starting.
I20250813 20:00:32.148949 17980 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.194:39745
I20250813 20:00:32.149138 18111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.194:39745 every 8 connection(s)
I20250813 20:00:32.152104 17980 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/info.pb
I20250813 20:00:32.160624 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 17980
I20250813 20:00:32.163674 11411 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
/tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.11.36.195:36881
--local_ip_for_outbound_sockets=127.11.36.195
--tserver_master_addrs=127.11.36.254:42849
--webserver_port=39145
--webserver_interface=127.11.36.195
--builtin_ntp_servers=127.11.36.212:36029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250813 20:00:32.231204 18004 log.cc:826] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Log is configured to *not* fsync() on all Append() calls
I20250813 20:00:32.239323 18112 heartbeater.cc:344] Connected to a master server at 127.11.36.254:42849
I20250813 20:00:32.239835 18112 heartbeater.cc:461] Registering TS with master...
I20250813 20:00:32.241035 18112 heartbeater.cc:507] Master 127.11.36.254:42849 requested a full tablet report, sending...
I20250813 20:00:32.244849 17789 ts_manager.cc:194] Registered new tserver with Master: ba454781df5f46cd99240cf8036588a1 (127.11.36.194:39745)
I20250813 20:00:32.247606 17789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.194:47691
I20250813 20:00:32.445000 17869 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 20:00:32.445971 17869 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Bootstrap complete.
I20250813 20:00:32.447615 17869 ts_tablet_manager.cc:1397] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Time spent bootstrapping tablet: real 3.455s user 3.315s sys 0.056s
I20250813 20:00:32.463162 17869 raft_consensus.cc:357] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:32.466033 17869 raft_consensus.cc:738] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c0c58e02e84a97ae87212df4e00ac0, State: Initialized, Role: FOLLOWER
I20250813 20:00:32.466852 17869 consensus_queue.cc:260] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:32.470227 17869 ts_tablet_manager.cc:1428] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Time spent starting tablet: real 0.022s user 0.025s sys 0.000s
W20250813 20:00:32.523859 18113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250813 20:00:32.524571 18113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250813 20:00:32.525413 18113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250813 20:00:32.557310 18113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250813 20:00:32.558135 18113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.11.36.195
I20250813 20:00:32.591843 18113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.11.36.212:36029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.11.36.195:36881
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.11.36.195
--webserver_port=39145
--tserver_master_addrs=127.11.36.254:42849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.11.36.195
--log_dir=/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 13 Aug 2025 19:43:20 UTC on 5fd53c4cbb9d
build id 7553
TSAN enabled
I20250813 20:00:32.593281 18113 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250813 20:00:32.594906 18113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250813 20:00:32.607342 18124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 20:00:33.251816 18112 heartbeater.cc:499] Master 127.11.36.254:42849 was elected leader, sending a full tablet report...
I20250813 20:00:34.050231 18130 raft_consensus.cc:491] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 20:00:34.051076 18130 raft_consensus.cc:513] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
W20250813 20:00:34.066097 17865 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.11.36.195:36881: connect: Connection refused (error 111)
I20250813 20:00:34.067711 18130 leader_election.cc:290] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881), ba454781df5f46cd99240cf8036588a1 (127.11.36.194:39745)
W20250813 20:00:34.078780 17865 leader_election.cc:336] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881): Network error: Client connection negotiation failed: client connection to 127.11.36.195:36881: connect: Connection refused (error 111)
I20250813 20:00:34.098404 18067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48d3298b28c1457f9aa9b7553cfcd0a4" candidate_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ba454781df5f46cd99240cf8036588a1" is_pre_election: true
W20250813 20:00:34.115715 17866 leader_election.cc:343] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ba454781df5f46cd99240cf8036588a1 (127.11.36.194:39745): Illegal state: must be running to vote when last-logged opid is not known
I20250813 20:00:34.116364 17866 leader_election.cc:304] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c0c58e02e84a97ae87212df4e00ac0; no voters: b25247ef9cff4bb48e98ed5ac8134eb3, ba454781df5f46cd99240cf8036588a1
I20250813 20:00:34.117940 18130 raft_consensus.cc:2747] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250813 20:00:32.608983 18125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250813 20:00:34.146045 18128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250813 20:00:34.149737 18113 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250813 20:00:34.150169 18126 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1537 milliseconds
I20250813 20:00:34.151386 18113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250813 20:00:34.164599 18113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250813 20:00:34.172299 18113 hybrid_clock.cc:648] HybridClock initialized: now 1755115234172213 us; error 99 us; skew 500 ppm
I20250813 20:00:34.173120 18113 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250813 20:00:34.179680 18113 webserver.cc:480] Webserver started at http://127.11.36.195:39145/ using document root <none> and password file <none>
I20250813 20:00:34.180589 18113 fs_manager.cc:362] Metadata directory not provided
I20250813 20:00:34.180814 18113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250813 20:00:34.188721 18113 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250813 20:00:34.193213 18139 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250813 20:00:34.194264 18113 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250813 20:00:34.194576 18113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data,/tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal
uuid: "b25247ef9cff4bb48e98ed5ac8134eb3"
format_stamp: "Formatted at 2025-08-13 20:00:16 on dist-test-slave-l098"
I20250813 20:00:34.196329 18113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250813 20:00:34.249913 18113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250813 20:00:34.251255 18113 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250813 20:00:34.251652 18113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250813 20:00:34.254161 18113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250813 20:00:34.260116 18146 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250813 20:00:34.270210 18113 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250813 20:00:34.270478 18113 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.002s sys 0.000s
I20250813 20:00:34.270798 18113 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250813 20:00:34.277415 18113 ts_tablet_manager.cc:610] Registered 1 tablets
I20250813 20:00:34.277709 18113 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.000s sys 0.007s
I20250813 20:00:34.278002 18146 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Bootstrap starting.
I20250813 20:00:34.472360 18113 rpc_server.cc:307] RPC server started. Bound to: 127.11.36.195:36881
I20250813 20:00:34.472488 18252 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.11.36.195:36881 every 8 connection(s)
I20250813 20:00:34.475848 18113 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/info.pb
I20250813 20:00:34.478240 11411 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu as pid 18113
I20250813 20:00:34.559408 18253 heartbeater.cc:344] Connected to a master server at 127.11.36.254:42849
I20250813 20:00:34.559887 18253 heartbeater.cc:461] Registering TS with master...
I20250813 20:00:34.561035 18253 heartbeater.cc:507] Master 127.11.36.254:42849 requested a full tablet report, sending...
I20250813 20:00:34.565075 17789 ts_manager.cc:194] Registered new tserver with Master: b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881)
I20250813 20:00:34.567880 17789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.11.36.195:34241
I20250813 20:00:34.570353 11411 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250813 20:00:34.597580 18146 log.cc:826] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Log is configured to *not* fsync() on all Append() calls
I20250813 20:00:34.620137 18004 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 20:00:34.620807 18004 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Bootstrap complete.
I20250813 20:00:34.622028 18004 ts_tablet_manager.cc:1397] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Time spent bootstrapping tablet: real 2.686s user 2.576s sys 0.088s
I20250813 20:00:34.626639 18004 raft_consensus.cc:357] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:34.628590 18004 raft_consensus.cc:738] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba454781df5f46cd99240cf8036588a1, State: Initialized, Role: FOLLOWER
I20250813 20:00:34.629217 18004 consensus_queue.cc:260] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:34.631963 18004 ts_tablet_manager.cc:1428] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20250813 20:00:35.571679 18253 heartbeater.cc:499] Master 127.11.36.254:42849 was elected leader, sending a full tablet report...
I20250813 20:00:35.965626 18267 raft_consensus.cc:491] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 20:00:35.965977 18267 raft_consensus.cc:513] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:35.967276 18267 leader_election.cc:290] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881), ba454781df5f46cd99240cf8036588a1 (127.11.36.194:39745)
I20250813 20:00:35.968333 18067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48d3298b28c1457f9aa9b7553cfcd0a4" candidate_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ba454781df5f46cd99240cf8036588a1" is_pre_election: true
I20250813 20:00:35.968981 18067 raft_consensus.cc:2466] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c0c58e02e84a97ae87212df4e00ac0 in term 1.
I20250813 20:00:35.970052 17866 leader_election.cc:304] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba454781df5f46cd99240cf8036588a1, c6c0c58e02e84a97ae87212df4e00ac0; no voters:
I20250813 20:00:35.970682 18267 raft_consensus.cc:2802] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250813 20:00:35.971030 18267 raft_consensus.cc:491] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 20:00:35.971319 18267 raft_consensus.cc:3058] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Advancing to term 2
I20250813 20:00:35.978371 18267 raft_consensus.cc:513] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:35.980394 18267 leader_election.cc:290] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [CANDIDATE]: Term 2 election: Requested vote from peers b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881), ba454781df5f46cd99240cf8036588a1 (127.11.36.194:39745)
I20250813 20:00:35.981003 18067 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48d3298b28c1457f9aa9b7553cfcd0a4" candidate_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "ba454781df5f46cd99240cf8036588a1"
I20250813 20:00:35.981474 18067 raft_consensus.cc:3058] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 1 FOLLOWER]: Advancing to term 2
I20250813 20:00:35.982342 18208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48d3298b28c1457f9aa9b7553cfcd0a4" candidate_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" is_pre_election: true
I20250813 20:00:35.982821 18207 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48d3298b28c1457f9aa9b7553cfcd0a4" candidate_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3"
I20250813 20:00:35.989765 18067 raft_consensus.cc:2466] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6c0c58e02e84a97ae87212df4e00ac0 in term 2.
I20250813 20:00:35.990602 17866 leader_election.cc:304] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba454781df5f46cd99240cf8036588a1, c6c0c58e02e84a97ae87212df4e00ac0; no voters:
W20250813 20:00:35.991289 17865 leader_election.cc:343] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881): Illegal state: must be running to vote when last-logged opid is not known
I20250813 20:00:35.991324 18267 raft_consensus.cc:2802] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 2 FOLLOWER]: Leader election won for term 2
W20250813 20:00:35.992138 17865 leader_election.cc:343] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881): Illegal state: must be running to vote when last-logged opid is not known
I20250813 20:00:35.992935 18267 raft_consensus.cc:695] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 2 LEADER]: Becoming Leader. State: Replica: c6c0c58e02e84a97ae87212df4e00ac0, State: Running, Role: LEADER
I20250813 20:00:35.993834 18267 consensus_queue.cc:237] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:36.004151 17788 catalog_manager.cc:5582] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 reported cstate change: term changed from 0 to 2, leader changed from <none> to c6c0c58e02e84a97ae87212df4e00ac0 (127.11.36.193), VOTER b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195) added, VOTER ba454781df5f46cd99240cf8036588a1 (127.11.36.194) added, VOTER c6c0c58e02e84a97ae87212df4e00ac0 (127.11.36.193) added. New cstate: current_term: 2 leader_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } health_report { overall_health: UNKNOWN } } }
I20250813 20:00:36.411314 18146 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250813 20:00:36.412315 18146 tablet_bootstrap.cc:492] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Bootstrap complete.
I20250813 20:00:36.413913 18146 ts_tablet_manager.cc:1397] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Time spent bootstrapping tablet: real 2.136s user 2.048s sys 0.076s
I20250813 20:00:36.420778 18146 raft_consensus.cc:357] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:36.423333 18146 raft_consensus.cc:738] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25247ef9cff4bb48e98ed5ac8134eb3, State: Initialized, Role: FOLLOWER
I20250813 20:00:36.424132 18146 consensus_queue.cc:260] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } }
I20250813 20:00:36.427109 18146 ts_tablet_manager.cc:1428] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
W20250813 20:00:36.437737 11411 scanner-internal.cc:458] Time spent opening tablet: real 1.835s user 0.008s sys 0.000s
I20250813 20:00:36.516412 18208 raft_consensus.cc:3058] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 1 FOLLOWER]: Advancing to term 2
I20250813 20:00:36.525206 18208 raft_consensus.cc:1273] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 2 FOLLOWER]: Refusing update from remote peer c6c0c58e02e84a97ae87212df4e00ac0: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250813 20:00:36.526854 18281 consensus_queue.cc:1035] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250813 20:00:36.557024 18067 raft_consensus.cc:1273] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 2 FOLLOWER]: Refusing update from remote peer c6c0c58e02e84a97ae87212df4e00ac0: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250813 20:00:36.564373 18283 consensus_queue.cc:1035] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
I20250813 20:00:36.604943 17931 consensus_queue.cc:237] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } }
I20250813 20:00:36.609578 18208 raft_consensus.cc:1273] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 2 FOLLOWER]: Refusing update from remote peer c6c0c58e02e84a97ae87212df4e00ac0: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250813 20:00:36.610905 18283 consensus_queue.cc:1035] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250813 20:00:36.616034 18287 raft_consensus.cc:2953] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER ba454781df5f46cd99240cf8036588a1 (127.11.36.194) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } }
I20250813 20:00:36.617496 18208 raft_consensus.cc:2953] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER ba454781df5f46cd99240cf8036588a1 (127.11.36.194) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } }
I20250813 20:00:36.625973 17775 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 48d3298b28c1457f9aa9b7553cfcd0a4 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250813 20:00:36.629118 17788 catalog_manager.cc:5582] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 reported cstate change: config changed from index -1 to 207, VOTER ba454781df5f46cd99240cf8036588a1 (127.11.36.194) evicted. New cstate: current_term: 2 leader_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } health_report { overall_health: HEALTHY } } }
I20250813 20:00:36.645294 17931 consensus_queue.cc:237] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } }
I20250813 20:00:36.651129 18287 raft_consensus.cc:2953] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } } }
I20250813 20:00:36.660478 17775 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 48d3298b28c1457f9aa9b7553cfcd0a4 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250813 20:00:36.660862 18047 tablet_service.cc:1515] Processing DeleteTablet for tablet 48d3298b28c1457f9aa9b7553cfcd0a4 with delete_type TABLET_DATA_TOMBSTONED (TS ba454781df5f46cd99240cf8036588a1 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:52286
I20250813 20:00:36.666965 17788 catalog_manager.cc:5582] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 reported cstate change: config changed from index 207 to 208, VOTER b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195) evicted. New cstate: current_term: 2 leader_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } health_report { overall_health: HEALTHY } } }
I20250813 20:00:36.676806 18296 tablet_replica.cc:331] stopping tablet replica
I20250813 20:00:36.677676 18296 raft_consensus.cc:2241] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250813 20:00:36.678289 18296 raft_consensus.cc:2270] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250813 20:00:36.698879 18188 tablet_service.cc:1515] Processing DeleteTablet for tablet 48d3298b28c1457f9aa9b7553cfcd0a4 with delete_type TABLET_DATA_TOMBSTONED (TS b25247ef9cff4bb48e98ed5ac8134eb3 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:34944
I20250813 20:00:36.699944 18296 ts_tablet_manager.cc:1905] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250813 20:00:36.710393 18298 tablet_replica.cc:331] stopping tablet replica
I20250813 20:00:36.711365 18298 raft_consensus.cc:2241] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250813 20:00:36.712194 18298 raft_consensus.cc:2270] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250813 20:00:36.717665 18296 ts_tablet_manager.cc:1918] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250813 20:00:36.718098 18296 log.cc:1199] T 48d3298b28c1457f9aa9b7553cfcd0a4 P ba454781df5f46cd99240cf8036588a1: Deleting WAL directory at /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/wal/wals/48d3298b28c1457f9aa9b7553cfcd0a4
I20250813 20:00:36.719910 17776 catalog_manager.cc:4928] TS ba454781df5f46cd99240cf8036588a1 (127.11.36.194:39745): tablet 48d3298b28c1457f9aa9b7553cfcd0a4 (table pre_rebuild [id=627bdc5918184b8a826309aab13ed7a8]) successfully deleted
I20250813 20:00:36.729277 18298 ts_tablet_manager.cc:1905] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250813 20:00:36.740270 18298 ts_tablet_manager.cc:1918] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250813 20:00:36.740689 18298 log.cc:1199] T 48d3298b28c1457f9aa9b7553cfcd0a4 P b25247ef9cff4bb48e98ed5ac8134eb3: Deleting WAL directory at /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/wal/wals/48d3298b28c1457f9aa9b7553cfcd0a4
I20250813 20:00:36.742110 17775 catalog_manager.cc:4928] TS b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881): tablet 48d3298b28c1457f9aa9b7553cfcd0a4 (table pre_rebuild [id=627bdc5918184b8a826309aab13ed7a8]) successfully deleted
I20250813 20:00:37.077781 18047 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 20:00:37.078168 18188 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 20:00:37.079703 17911 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ed87e9d4a7084a3aa2c026ae1c329ea1 | 127.11.36.254:42849 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.11.36.212:36029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
b25247ef9cff4bb48e98ed5ac8134eb3 | 127.11.36.195:36881 | HEALTHY | <none> | 0 | 0
ba454781df5f46cd99240cf8036588a1 | 127.11.36.194:39745 | HEALTHY | <none> | 0 | 0
c6c0c58e02e84a97ae87212df4e00ac0 | 127.11.36.193:37389 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.11.36.193 | experimental | 127.11.36.193:37389
local_ip_for_outbound_sockets | 127.11.36.194 | experimental | 127.11.36.194:39745
local_ip_for_outbound_sockets | 127.11.36.195 | experimental | 127.11.36.195:36881
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/info.pb | hidden | 127.11.36.193:37389
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/info.pb | hidden | 127.11.36.194:39745
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/info.pb | hidden | 127.11.36.195:36881
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.11.36.212:36029 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250813 20:00:37.430083 11411 log_verifier.cc:126] Checking tablet 48d3298b28c1457f9aa9b7553cfcd0a4
I20250813 20:00:37.716578 11411 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 48d3298b28c1457f9aa9b7553cfcd0a4
I20250813 20:00:37.718940 17789 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40932:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250813 20:00:37.719391 17789 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40932:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250813 20:00:37.730778 17789 catalog_manager.cc:5869] T 00000000000000000000000000000000 P ed87e9d4a7084a3aa2c026ae1c329ea1: Sending DeleteTablet for 1 replicas of tablet 48d3298b28c1457f9aa9b7553cfcd0a4
I20250813 20:00:37.732415 11411 test_util.cc:276] Using random seed: -1672941532
I20250813 20:00:37.733104 17911 tablet_service.cc:1515] Processing DeleteTablet for tablet 48d3298b28c1457f9aa9b7553cfcd0a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-13 20:00:37 UTC) from {username='slave'} at 127.0.0.1:36678
I20250813 20:00:37.735419 18327 tablet_replica.cc:331] stopping tablet replica
I20250813 20:00:37.736356 18327 raft_consensus.cc:2241] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 2 LEADER]: Raft consensus shutting down.
I20250813 20:00:37.737320 18327 raft_consensus.cc:2270] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250813 20:00:37.769572 17789 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40974:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250813 20:00:37.772162 17789 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250813 20:00:37.784910 18327 ts_tablet_manager.cc:1905] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250813 20:00:37.795181 18047 tablet_service.cc:1468] Processing CreateTablet for tablet 306f434b89c94735993095a866829a57 (DEFAULT_TABLE table=post_rebuild [id=8a8d547f45eb4ff5b85445cc38e8713a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 20:00:37.795498 18188 tablet_service.cc:1468] Processing CreateTablet for tablet 306f434b89c94735993095a866829a57 (DEFAULT_TABLE table=post_rebuild [id=8a8d547f45eb4ff5b85445cc38e8713a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 20:00:37.796654 18047 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306f434b89c94735993095a866829a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 20:00:37.796820 18188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306f434b89c94735993095a866829a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 20:00:37.802548 17911 tablet_service.cc:1468] Processing CreateTablet for tablet 306f434b89c94735993095a866829a57 (DEFAULT_TABLE table=post_rebuild [id=8a8d547f45eb4ff5b85445cc38e8713a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250813 20:00:37.804261 17911 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306f434b89c94735993095a866829a57. 1 dirs total, 0 dirs full, 0 dirs failed
I20250813 20:00:37.825168 18334 tablet_bootstrap.cc:492] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3: Bootstrap starting.
I20250813 20:00:37.826187 18327 ts_tablet_manager.cc:1918] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250813 20:00:37.826977 18335 tablet_bootstrap.cc:492] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1: Bootstrap starting.
I20250813 20:00:37.830960 18327 log.cc:1199] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Deleting WAL directory at /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/wal/wals/48d3298b28c1457f9aa9b7553cfcd0a4
I20250813 20:00:37.831964 18334 tablet_bootstrap.cc:654] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3: Neither blocks nor log segments found. Creating new log.
I20250813 20:00:37.832301 18327 ts_tablet_manager.cc:1939] T 48d3298b28c1457f9aa9b7553cfcd0a4 P c6c0c58e02e84a97ae87212df4e00ac0: Deleting consensus metadata
I20250813 20:00:37.834514 18335 tablet_bootstrap.cc:654] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1: Neither blocks nor log segments found. Creating new log.
I20250813 20:00:37.837286 17775 catalog_manager.cc:4928] TS c6c0c58e02e84a97ae87212df4e00ac0 (127.11.36.193:37389): tablet 48d3298b28c1457f9aa9b7553cfcd0a4 (table pre_rebuild [id=627bdc5918184b8a826309aab13ed7a8]) successfully deleted
I20250813 20:00:37.841706 18336 tablet_bootstrap.cc:492] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0: Bootstrap starting.
I20250813 20:00:37.843338 18334 tablet_bootstrap.cc:492] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3: No bootstrap required, opened a new log
I20250813 20:00:37.843851 18334 ts_tablet_manager.cc:1397] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.009s
I20250813 20:00:37.846771 18334 raft_consensus.cc:357] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } }
I20250813 20:00:37.847457 18334 raft_consensus.cc:383] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 20:00:37.847888 18334 raft_consensus.cc:738] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b25247ef9cff4bb48e98ed5ac8134eb3, State: Initialized, Role: FOLLOWER
I20250813 20:00:37.848769 18334 consensus_queue.cc:260] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } }
I20250813 20:00:37.849977 18335 tablet_bootstrap.cc:492] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1: No bootstrap required, opened a new log
I20250813 20:00:37.850512 18335 ts_tablet_manager.cc:1397] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1: Time spent bootstrapping tablet: real 0.024s user 0.013s sys 0.005s
I20250813 20:00:37.853462 18336 tablet_bootstrap.cc:654] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0: Neither blocks nor log segments found. Creating new log.
I20250813 20:00:37.854442 18335 raft_consensus.cc:357] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } }
I20250813 20:00:37.855435 18335 raft_consensus.cc:383] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 20:00:37.855962 18335 raft_consensus.cc:738] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba454781df5f46cd99240cf8036588a1, State: Initialized, Role: FOLLOWER
I20250813 20:00:37.856768 18335 consensus_queue.cc:260] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } }
I20250813 20:00:37.858508 18334 ts_tablet_manager.cc:1428] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3: Time spent starting tablet: real 0.014s user 0.006s sys 0.006s
I20250813 20:00:37.876628 18336 tablet_bootstrap.cc:492] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0: No bootstrap required, opened a new log
I20250813 20:00:37.877138 18335 ts_tablet_manager.cc:1428] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1: Time spent starting tablet: real 0.026s user 0.016s sys 0.007s
I20250813 20:00:37.877207 18336 ts_tablet_manager.cc:1397] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0: Time spent bootstrapping tablet: real 0.036s user 0.013s sys 0.013s
I20250813 20:00:37.880165 18336 raft_consensus.cc:357] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } }
I20250813 20:00:37.881096 18336 raft_consensus.cc:383] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250813 20:00:37.881429 18336 raft_consensus.cc:738] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c0c58e02e84a97ae87212df4e00ac0, State: Initialized, Role: FOLLOWER
I20250813 20:00:37.882088 18336 consensus_queue.cc:260] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } }
I20250813 20:00:37.884742 18336 ts_tablet_manager.cc:1428] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0: Time spent starting tablet: real 0.007s user 0.005s sys 0.001s
W20250813 20:00:37.948220 17977 tablet.cc:2378] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250813 20:00:37.961043 18114 tablet.cc:2378] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 20:00:38.009425 18341 raft_consensus.cc:491] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250813 20:00:38.009959 18341 raft_consensus.cc:513] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } }
I20250813 20:00:38.012617 18341 leader_election.cc:290] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881), c6c0c58e02e84a97ae87212df4e00ac0 (127.11.36.193:37389)
I20250813 20:00:38.029773 18208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "306f434b89c94735993095a866829a57" candidate_uuid: "ba454781df5f46cd99240cf8036588a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" is_pre_election: true
I20250813 20:00:38.030450 18208 raft_consensus.cc:2466] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba454781df5f46cd99240cf8036588a1 in term 0.
I20250813 20:00:38.031652 18000 leader_election.cc:304] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b25247ef9cff4bb48e98ed5ac8134eb3, ba454781df5f46cd99240cf8036588a1; no voters:
I20250813 20:00:38.032418 18341 raft_consensus.cc:2802] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250813 20:00:38.032727 18341 raft_consensus.cc:491] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250813 20:00:38.033018 18341 raft_consensus.cc:3058] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 0 FOLLOWER]: Advancing to term 1
I20250813 20:00:38.037708 17931 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "306f434b89c94735993095a866829a57" candidate_uuid: "ba454781df5f46cd99240cf8036588a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" is_pre_election: true
I20250813 20:00:38.038575 17931 raft_consensus.cc:2466] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba454781df5f46cd99240cf8036588a1 in term 0.
I20250813 20:00:38.039268 18341 raft_consensus.cc:513] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } }
I20250813 20:00:38.040977 18341 leader_election.cc:290] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [CANDIDATE]: Term 1 election: Requested vote from peers b25247ef9cff4bb48e98ed5ac8134eb3 (127.11.36.195:36881), c6c0c58e02e84a97ae87212df4e00ac0 (127.11.36.193:37389)
I20250813 20:00:38.041626 18208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "306f434b89c94735993095a866829a57" candidate_uuid: "ba454781df5f46cd99240cf8036588a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3"
I20250813 20:00:38.041832 17931 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "306f434b89c94735993095a866829a57" candidate_uuid: "ba454781df5f46cd99240cf8036588a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c0c58e02e84a97ae87212df4e00ac0"
I20250813 20:00:38.042011 18208 raft_consensus.cc:3058] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 0 FOLLOWER]: Advancing to term 1
I20250813 20:00:38.042282 17931 raft_consensus.cc:3058] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0 [term 0 FOLLOWER]: Advancing to term 1
I20250813 20:00:38.045883 18208 raft_consensus.cc:2466] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba454781df5f46cd99240cf8036588a1 in term 1.
I20250813 20:00:38.046581 18000 leader_election.cc:304] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b25247ef9cff4bb48e98ed5ac8134eb3, ba454781df5f46cd99240cf8036588a1; no voters:
I20250813 20:00:38.047194 18341 raft_consensus.cc:2802] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 1 FOLLOWER]: Leader election won for term 1
I20250813 20:00:38.048447 17931 raft_consensus.cc:2466] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba454781df5f46cd99240cf8036588a1 in term 1.
I20250813 20:00:38.049388 18341 raft_consensus.cc:695] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [term 1 LEADER]: Becoming Leader. State: Replica: ba454781df5f46cd99240cf8036588a1, State: Running, Role: LEADER
I20250813 20:00:38.050163 18341 consensus_queue.cc:237] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } } peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } }
W20250813 20:00:38.052027 18256 tablet.cc:2378] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250813 20:00:38.058233 17789 catalog_manager.cc:5582] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba454781df5f46cd99240cf8036588a1 (127.11.36.194). New cstate: current_term: 1 leader_uuid: "ba454781df5f46cd99240cf8036588a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba454781df5f46cd99240cf8036588a1" member_type: VOTER last_known_addr { host: "127.11.36.194" port: 39745 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 } health_report { overall_health: UNKNOWN } } }
I20250813 20:00:38.356730 17931 raft_consensus.cc:1273] T 306f434b89c94735993095a866829a57 P c6c0c58e02e84a97ae87212df4e00ac0 [term 1 FOLLOWER]: Refusing update from remote peer ba454781df5f46cd99240cf8036588a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250813 20:00:38.357273 18208 raft_consensus.cc:1273] T 306f434b89c94735993095a866829a57 P b25247ef9cff4bb48e98ed5ac8134eb3 [term 1 FOLLOWER]: Refusing update from remote peer ba454781df5f46cd99240cf8036588a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250813 20:00:38.358541 18341 consensus_queue.cc:1035] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b25247ef9cff4bb48e98ed5ac8134eb3" member_type: VOTER last_known_addr { host: "127.11.36.195" port: 36881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250813 20:00:38.359326 18347 consensus_queue.cc:1035] T 306f434b89c94735993095a866829a57 P ba454781df5f46cd99240cf8036588a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c0c58e02e84a97ae87212df4e00ac0" member_type: VOTER last_known_addr { host: "127.11.36.193" port: 37389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250813 20:00:42.940522 18047 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250813 20:00:42.942452 18188 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250813 20:00:42.951342 17911 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
ed87e9d4a7084a3aa2c026ae1c329ea1 | 127.11.36.254:42849 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.11.36.212:36029 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
b25247ef9cff4bb48e98ed5ac8134eb3 | 127.11.36.195:36881 | HEALTHY | <none> | 0 | 0
ba454781df5f46cd99240cf8036588a1 | 127.11.36.194:39745 | HEALTHY | <none> | 1 | 0
c6c0c58e02e84a97ae87212df4e00ac0 | 127.11.36.193:37389 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.11.36.193 | experimental | 127.11.36.193:37389
local_ip_for_outbound_sockets | 127.11.36.194 | experimental | 127.11.36.194:39745
local_ip_for_outbound_sockets | 127.11.36.195 | experimental | 127.11.36.195:36881
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-0/data/info.pb | hidden | 127.11.36.193:37389
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-1/data/info.pb | hidden | 127.11.36.194:39745
server_dump_info_path | /tmp/dist-test-taskEVEnVz/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755115061406653-11411-0/minicluster-data/ts-2/data/info.pb | hidden | 127.11.36.195:36881
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.11.36.212:36029 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250813 20:00:43.161859 11411 log_verifier.cc:126] Checking tablet 306f434b89c94735993095a866829a57
I20250813 20:00:43.922336 11411 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 306f434b89c94735993095a866829a57
I20250813 20:00:43.923130 11411 log_verifier.cc:126] Checking tablet 48d3298b28c1457f9aa9b7553cfcd0a4
I20250813 20:00:43.923382 11411 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 48d3298b28c1457f9aa9b7553cfcd0a4
I20250813 20:00:43.947722 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 17826
I20250813 20:00:43.989094 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 17980
I20250813 20:00:44.028733 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 18113
I20250813 20:00:44.064208 11411 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskEVEnVz/build/tsan/bin/kudu with pid 17756
2025-08-13T20:00:44Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (35074 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (35074 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (182642 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST