Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250625 07:57:12.572052 5461 test_util.cc:276] Using random seed: -890675186
W20250625 07:57:13.706245 5461 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.092s user 0.506s sys 0.582s
W20250625 07:57:13.706578 5461 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.092s user 0.506s sys 0.582s
I20250625 07:57:13.715566 5461 ts_itest-base.cc:115] Starting cluster with:
I20250625 07:57:13.715749 5461 ts_itest-base.cc:116] --------------
I20250625 07:57:13.715890 5461 ts_itest-base.cc:117] 4 tablet servers
I20250625 07:57:13.716058 5461 ts_itest-base.cc:118] 3 replicas per TS
I20250625 07:57:13.716224 5461 ts_itest-base.cc:119] --------------
2025-06-25T07:57:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T07:57:13Z Disabled control of system clock
I20250625 07:57:13.750922 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:41951
--webserver_interface=127.5.85.126
--webserver_port=0
--builtin_ntp_servers=127.5.85.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:41951 with env {}
W20250625 07:57:14.051807 5475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:14.052378 5475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:14.052822 5475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:14.083609 5475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:57:14.083910 5475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:14.084157 5475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:57:14.084384 5475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:57:14.119225 5475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:41951
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:41951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:14.120646 5475 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:14.122323 5475 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:14.138105 5482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:14.138095 5481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:14.139261 5484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:15.232756 5483 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:57:15.232867 5475 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:57:15.237103 5475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:15.240001 5475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:15.241427 5475 hybrid_clock.cc:648] HybridClock initialized: now 1750838235241388 us; error 56 us; skew 500 ppm
I20250625 07:57:15.242179 5475 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:15.253249 5475 webserver.cc:469] Webserver started at http://127.5.85.126:45151/ using document root <none> and password file <none>
I20250625 07:57:15.254163 5475 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:15.254362 5475 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:15.254820 5475 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:15.262777 5475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "4272391e062e4e14b01ab1a5b5edec15"
format_stamp: "Formatted at 2025-06-25 07:57:15 on dist-test-slave-8zlz"
I20250625 07:57:15.263856 5475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "4272391e062e4e14b01ab1a5b5edec15"
format_stamp: "Formatted at 2025-06-25 07:57:15 on dist-test-slave-8zlz"
I20250625 07:57:15.271307 5475 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250625 07:57:15.276880 5491 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:15.277971 5475 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250625 07:57:15.278402 5475 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4272391e062e4e14b01ab1a5b5edec15"
format_stamp: "Formatted at 2025-06-25 07:57:15 on dist-test-slave-8zlz"
I20250625 07:57:15.278818 5475 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:15.379489 5475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:15.381023 5475 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:15.381495 5475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:15.470785 5475 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:41951
I20250625 07:57:15.470858 5542 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:41951 every 8 connection(s)
I20250625 07:57:15.473392 5475 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 07:57:15.478739 5543 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:15.484248 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 5475
I20250625 07:57:15.484678 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 07:57:15.498934 5543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15: Bootstrap starting.
I20250625 07:57:15.505028 5543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:15.507254 5543 log.cc:826] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:15.513171 5543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15: No bootstrap required, opened a new log
I20250625 07:57:15.534718 5543 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } }
I20250625 07:57:15.535419 5543 raft_consensus.cc:383] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:15.535620 5543 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4272391e062e4e14b01ab1a5b5edec15, State: Initialized, Role: FOLLOWER
I20250625 07:57:15.536270 5543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } }
I20250625 07:57:15.536763 5543 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:57:15.537043 5543 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:57:15.537343 5543 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:15.541882 5543 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } }
I20250625 07:57:15.542659 5543 leader_election.cc:304] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4272391e062e4e14b01ab1a5b5edec15; no voters:
I20250625 07:57:15.544732 5543 leader_election.cc:290] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:57:15.545333 5548 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:57:15.548605 5548 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 1 LEADER]: Becoming Leader. State: Replica: 4272391e062e4e14b01ab1a5b5edec15, State: Running, Role: LEADER
I20250625 07:57:15.549960 5548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } }
I20250625 07:57:15.551960 5543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:57:15.564792 5550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4272391e062e4e14b01ab1a5b5edec15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } } }
I20250625 07:57:15.565369 5550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [sys.catalog]: This master's current role is: LEADER
I20250625 07:57:15.566255 5549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4272391e062e4e14b01ab1a5b5edec15. Latest consensus state: current_term: 1 leader_uuid: "4272391e062e4e14b01ab1a5b5edec15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } } }
I20250625 07:57:15.566893 5549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [sys.catalog]: This master's current role is: LEADER
I20250625 07:57:15.570670 5558 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:57:15.583329 5558 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:57:15.600323 5558 catalog_manager.cc:1349] Generated new cluster ID: c5cbada092964b60a648a84a2552d2dc
I20250625 07:57:15.600595 5558 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:57:15.623165 5558 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 07:57:15.624671 5558 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:57:15.641060 5558 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15: Generated new TSK 0
I20250625 07:57:15.641988 5558 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 07:57:15.665357 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:0
--local_ip_for_outbound_sockets=127.5.85.65
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:41951
--builtin_ntp_servers=127.5.85.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 07:57:15.982232 5567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:15.982749 5567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:15.983243 5567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:16.014544 5567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:16.015425 5567 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:57:16.050273 5567 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:41951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:16.051586 5567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:16.053246 5567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:16.072386 5573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:16.073886 5576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:16.072633 5574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:17.252140 5575 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1178 milliseconds
I20250625 07:57:17.252256 5567 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:57:17.253468 5567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:17.255893 5567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:17.257341 5567 hybrid_clock.cc:648] HybridClock initialized: now 1750838237257282 us; error 70 us; skew 500 ppm
I20250625 07:57:17.258155 5567 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:17.265259 5567 webserver.cc:469] Webserver started at http://127.5.85.65:37003/ using document root <none> and password file <none>
I20250625 07:57:17.266144 5567 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:17.266345 5567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:17.266824 5567 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:17.271588 5567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "726e287ed0c74cffb6e4812fb95c19f2"
format_stamp: "Formatted at 2025-06-25 07:57:17 on dist-test-slave-8zlz"
I20250625 07:57:17.272699 5567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "726e287ed0c74cffb6e4812fb95c19f2"
format_stamp: "Formatted at 2025-06-25 07:57:17 on dist-test-slave-8zlz"
I20250625 07:57:17.280570 5567 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250625 07:57:17.287372 5583 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:17.288508 5567 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.004s
I20250625 07:57:17.288849 5567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "726e287ed0c74cffb6e4812fb95c19f2"
format_stamp: "Formatted at 2025-06-25 07:57:17 on dist-test-slave-8zlz"
I20250625 07:57:17.289183 5567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:17.338927 5567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:17.340420 5567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:17.340853 5567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:17.344834 5567 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:57:17.349697 5567 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:57:17.349898 5567 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:17.350149 5567 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:57:17.350307 5567 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:17.536800 5567 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:36697
I20250625 07:57:17.536980 5695 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:36697 every 8 connection(s)
I20250625 07:57:17.539398 5567 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 07:57:17.543475 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 5567
I20250625 07:57:17.544005 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 07:57:17.558426 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.66:0
--local_ip_for_outbound_sockets=127.5.85.66
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:41951
--builtin_ntp_servers=127.5.85.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:57:17.583721 5696 heartbeater.cc:344] Connected to a master server at 127.5.85.126:41951
I20250625 07:57:17.584228 5696 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:17.585489 5696 heartbeater.cc:507] Master 127.5.85.126:41951 requested a full tablet report, sending...
I20250625 07:57:17.588550 5508 ts_manager.cc:194] Registered new tserver with Master: 726e287ed0c74cffb6e4812fb95c19f2 (127.5.85.65:36697)
I20250625 07:57:17.591292 5508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:39997
W20250625 07:57:17.873216 5700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:17.873724 5700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:17.874218 5700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:17.905239 5700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:17.906090 5700 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.66
I20250625 07:57:17.942919 5700 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:41951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.66
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:17.944236 5700 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:17.945887 5700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:17.962684 5707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:18.595564 5696 heartbeater.cc:499] Master 127.5.85.126:41951 was elected leader, sending a full tablet report...
W20250625 07:57:17.963151 5709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:17.963594 5706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:19.055676 5708 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:57:19.055763 5700 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:57:19.059609 5700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:19.061758 5700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:19.063125 5700 hybrid_clock.cc:648] HybridClock initialized: now 1750838239063072 us; error 54 us; skew 500 ppm
I20250625 07:57:19.063937 5700 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:19.070411 5700 webserver.cc:469] Webserver started at http://127.5.85.66:46559/ using document root <none> and password file <none>
I20250625 07:57:19.071382 5700 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:19.071609 5700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:19.072052 5700 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:19.076508 5700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "ca9c4b589f6f45f89952373beea90665"
format_stamp: "Formatted at 2025-06-25 07:57:19 on dist-test-slave-8zlz"
I20250625 07:57:19.077625 5700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "ca9c4b589f6f45f89952373beea90665"
format_stamp: "Formatted at 2025-06-25 07:57:19 on dist-test-slave-8zlz"
I20250625 07:57:19.084844 5700 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250625 07:57:19.091223 5716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:19.092339 5700 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250625 07:57:19.092705 5700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "ca9c4b589f6f45f89952373beea90665"
format_stamp: "Formatted at 2025-06-25 07:57:19 on dist-test-slave-8zlz"
I20250625 07:57:19.093030 5700 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:19.154021 5700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:19.155822 5700 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:19.156256 5700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:19.159166 5700 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:57:19.163642 5700 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:57:19.163848 5700 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:19.164098 5700 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:57:19.164242 5700 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:19.311869 5700 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.66:45235
I20250625 07:57:19.311960 5828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.66:45235 every 8 connection(s)
I20250625 07:57:19.314311 5700 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 07:57:19.321352 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 5700
I20250625 07:57:19.321862 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 07:57:19.328564 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.67:0
--local_ip_for_outbound_sockets=127.5.85.67
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:41951
--builtin_ntp_servers=127.5.85.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:57:19.337679 5829 heartbeater.cc:344] Connected to a master server at 127.5.85.126:41951
I20250625 07:57:19.338733 5829 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:19.339972 5829 heartbeater.cc:507] Master 127.5.85.126:41951 requested a full tablet report, sending...
I20250625 07:57:19.342314 5508 ts_manager.cc:194] Registered new tserver with Master: ca9c4b589f6f45f89952373beea90665 (127.5.85.66:45235)
I20250625 07:57:19.343582 5508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.66:34311
W20250625 07:57:19.637387 5833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:19.637873 5833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:19.638383 5833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:19.669039 5833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:19.669869 5833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.67
I20250625 07:57:19.704880 5833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:41951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.67
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:19.706157 5833 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:19.707814 5833 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:19.724529 5840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:20.347371 5829 heartbeater.cc:499] Master 127.5.85.126:41951 was elected leader, sending a full tablet report...
W20250625 07:57:19.724611 5839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:19.725730 5842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:20.804594 5841 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:57:20.804699 5833 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:57:20.808703 5833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:20.811748 5833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:20.813194 5833 hybrid_clock.cc:648] HybridClock initialized: now 1750838240813128 us; error 79 us; skew 500 ppm
I20250625 07:57:20.814002 5833 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:20.823632 5833 webserver.cc:469] Webserver started at http://127.5.85.67:37635/ using document root <none> and password file <none>
I20250625 07:57:20.824535 5833 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:20.824740 5833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:20.825172 5833 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:20.829839 5833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0b10743963c243339440eed8d461304c"
format_stamp: "Formatted at 2025-06-25 07:57:20 on dist-test-slave-8zlz"
I20250625 07:57:20.830910 5833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0b10743963c243339440eed8d461304c"
format_stamp: "Formatted at 2025-06-25 07:57:20 on dist-test-slave-8zlz"
I20250625 07:57:20.837931 5833 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250625 07:57:20.843201 5849 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:20.844139 5833 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250625 07:57:20.844439 5833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0b10743963c243339440eed8d461304c"
format_stamp: "Formatted at 2025-06-25 07:57:20 on dist-test-slave-8zlz"
I20250625 07:57:20.844736 5833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:20.917065 5833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:20.918495 5833 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:20.918910 5833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:20.921834 5833 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:57:20.925629 5833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:57:20.925833 5833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:20.926086 5833 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:57:20.926244 5833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:21.068384 5833 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.67:46403
I20250625 07:57:21.068468 5961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.67:46403 every 8 connection(s)
I20250625 07:57:21.071053 5833 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 07:57:21.078620 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 5833
I20250625 07:57:21.079113 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 07:57:21.084708 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.68:0
--local_ip_for_outbound_sockets=127.5.85.68
--webserver_interface=127.5.85.68
--webserver_port=0
--tserver_master_addrs=127.5.85.126:41951
--builtin_ntp_servers=127.5.85.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:57:21.092722 5962 heartbeater.cc:344] Connected to a master server at 127.5.85.126:41951
I20250625 07:57:21.093240 5962 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:21.094494 5962 heartbeater.cc:507] Master 127.5.85.126:41951 requested a full tablet report, sending...
I20250625 07:57:21.096956 5508 ts_manager.cc:194] Registered new tserver with Master: 0b10743963c243339440eed8d461304c (127.5.85.67:46403)
I20250625 07:57:21.098572 5508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.67:41831
W20250625 07:57:21.396688 5966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:21.397161 5966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:21.397626 5966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:21.430414 5966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:21.431326 5966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.68
I20250625 07:57:21.465937 5966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.85.68
--webserver_port=0
--tserver_master_addrs=127.5.85.126:41951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.68
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:21.467209 5966 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:21.468982 5966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:21.488787 5973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:22.102892 5962 heartbeater.cc:499] Master 127.5.85.126:41951 was elected leader, sending a full tablet report...
W20250625 07:57:22.887516 5971 debug-util.cc:398] Leaking SignalData structure 0x7b08000184e0 after lost signal to thread 5966
W20250625 07:57:23.019277 5971 kernel_stack_watchdog.cc:198] Thread 5966 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 396ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250625 07:57:21.488775 5972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:23.020778 5974 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1532 milliseconds
W20250625 07:57:23.024902 5966 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.537s user 0.002s sys 0.005s
W20250625 07:57:23.025218 5966 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.537s user 0.002s sys 0.005s
W20250625 07:57:23.026885 5976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:23.026894 5966 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:57:23.028116 5966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:23.030524 5966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:23.031982 5966 hybrid_clock.cc:648] HybridClock initialized: now 1750838243031930 us; error 69 us; skew 500 ppm
I20250625 07:57:23.032775 5966 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:23.039279 5966 webserver.cc:469] Webserver started at http://127.5.85.68:35925/ using document root <none> and password file <none>
I20250625 07:57:23.040230 5966 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:23.040447 5966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:23.040908 5966 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:23.045331 5966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "afcd783b188d46f1a5934c097532c907"
format_stamp: "Formatted at 2025-06-25 07:57:23 on dist-test-slave-8zlz"
I20250625 07:57:23.046519 5966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "afcd783b188d46f1a5934c097532c907"
format_stamp: "Formatted at 2025-06-25 07:57:23 on dist-test-slave-8zlz"
I20250625 07:57:23.054116 5966 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250625 07:57:23.061160 5983 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:23.062248 5966 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250625 07:57:23.062592 5966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "afcd783b188d46f1a5934c097532c907"
format_stamp: "Formatted at 2025-06-25 07:57:23 on dist-test-slave-8zlz"
I20250625 07:57:23.063005 5966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:23.132601 5966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:23.134027 5966 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:23.134438 5966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:23.137245 5966 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:57:23.141702 5966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:57:23.141902 5966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:23.142098 5966 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:57:23.142227 5966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:23.291246 5966 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.68:39635
I20250625 07:57:23.291347 6095 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.68:39635 every 8 connection(s)
I20250625 07:57:23.293709 5966 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250625 07:57:23.302846 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 5966
I20250625 07:57:23.303427 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250625 07:57:23.316354 6096 heartbeater.cc:344] Connected to a master server at 127.5.85.126:41951
I20250625 07:57:23.316901 6096 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:23.318244 6096 heartbeater.cc:507] Master 127.5.85.126:41951 requested a full tablet report, sending...
I20250625 07:57:23.320669 5508 ts_manager.cc:194] Registered new tserver with Master: afcd783b188d46f1a5934c097532c907 (127.5.85.68:39635)
I20250625 07:57:23.322153 5508 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.68:57809
I20250625 07:57:23.325580 5461 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250625 07:57:23.369161 5508 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45924:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250625 07:57:23.438510 5764 tablet_service.cc:1468] Processing CreateTablet for tablet a851a0efcec84427a4d2362d000a8f58 (DEFAULT_TABLE table=TestTable [id=c2ee5f408cce44bea42accbe0d1470a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:57:23.440321 5764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a851a0efcec84427a4d2362d000a8f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:23.441334 5897 tablet_service.cc:1468] Processing CreateTablet for tablet a851a0efcec84427a4d2362d000a8f58 (DEFAULT_TABLE table=TestTable [id=c2ee5f408cce44bea42accbe0d1470a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:57:23.443709 5897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a851a0efcec84427a4d2362d000a8f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:23.448002 6031 tablet_service.cc:1468] Processing CreateTablet for tablet a851a0efcec84427a4d2362d000a8f58 (DEFAULT_TABLE table=TestTable [id=c2ee5f408cce44bea42accbe0d1470a1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:57:23.449841 6031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a851a0efcec84427a4d2362d000a8f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:23.466594 6115 tablet_bootstrap.cc:492] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c: Bootstrap starting.
I20250625 07:57:23.470224 6116 tablet_bootstrap.cc:492] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665: Bootstrap starting.
I20250625 07:57:23.474997 6115 tablet_bootstrap.cc:654] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:23.477800 6115 log.cc:826] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:23.479218 6116 tablet_bootstrap.cc:654] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:23.481828 6116 log.cc:826] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:23.486305 6117 tablet_bootstrap.cc:492] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907: Bootstrap starting.
I20250625 07:57:23.490525 6116 tablet_bootstrap.cc:492] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665: No bootstrap required, opened a new log
I20250625 07:57:23.491068 6116 ts_tablet_manager.cc:1397] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.004s
I20250625 07:57:23.490513 6115 tablet_bootstrap.cc:492] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c: No bootstrap required, opened a new log
I20250625 07:57:23.491521 6115 ts_tablet_manager.cc:1397] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c: Time spent bootstrapping tablet: real 0.025s user 0.017s sys 0.005s
I20250625 07:57:23.495429 6117 tablet_bootstrap.cc:654] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:23.497742 6117 log.cc:826] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:23.510056 6117 tablet_bootstrap.cc:492] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907: No bootstrap required, opened a new log
I20250625 07:57:23.511250 6117 ts_tablet_manager.cc:1397] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.016s
I20250625 07:57:23.516683 6116 raft_consensus.cc:357] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:23.517805 6116 raft_consensus.cc:383] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:23.518143 6116 raft_consensus.cc:738] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca9c4b589f6f45f89952373beea90665, State: Initialized, Role: FOLLOWER
I20250625 07:57:23.519097 6116 consensus_queue.cc:260] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:23.519691 6115 raft_consensus.cc:357] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:23.520581 6115 raft_consensus.cc:383] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:23.520937 6115 raft_consensus.cc:738] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b10743963c243339440eed8d461304c, State: Initialized, Role: FOLLOWER
I20250625 07:57:23.521967 6115 consensus_queue.cc:260] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:23.533316 6116 ts_tablet_manager.cc:1428] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665: Time spent starting tablet: real 0.042s user 0.025s sys 0.016s
I20250625 07:57:23.543365 6122 raft_consensus.cc:491] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:57:23.547106 6122 raft_consensus.cc:513] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:23.551749 6117 raft_consensus.cc:357] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:23.552690 6117 raft_consensus.cc:383] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:23.553020 6117 raft_consensus.cc:738] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afcd783b188d46f1a5934c097532c907, State: Initialized, Role: FOLLOWER
I20250625 07:57:23.554517 6117 consensus_queue.cc:260] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:23.545377 6115 ts_tablet_manager.cc:1428] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c: Time spent starting tablet: real 0.054s user 0.027s sys 0.012s
I20250625 07:57:23.564141 6122 leader_election.cc:290] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ca9c4b589f6f45f89952373beea90665 (127.5.85.66:45235), afcd783b188d46f1a5934c097532c907 (127.5.85.68:39635)
I20250625 07:57:23.566440 6096 heartbeater.cc:499] Master 127.5.85.126:41951 was elected leader, sending a full tablet report...
I20250625 07:57:23.567986 6117 ts_tablet_manager.cc:1428] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907: Time spent starting tablet: real 0.056s user 0.035s sys 0.007s
W20250625 07:57:23.573124 5830 tablet.cc:2378] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 07:57:23.577890 5963 tablet.cc:2378] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:57:23.579174 6051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a851a0efcec84427a4d2362d000a8f58" candidate_uuid: "0b10743963c243339440eed8d461304c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afcd783b188d46f1a5934c097532c907" is_pre_election: true
I20250625 07:57:23.579403 5784 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a851a0efcec84427a4d2362d000a8f58" candidate_uuid: "0b10743963c243339440eed8d461304c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9c4b589f6f45f89952373beea90665" is_pre_election: true
I20250625 07:57:23.579795 6051 raft_consensus.cc:2466] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b10743963c243339440eed8d461304c in term 0.
I20250625 07:57:23.580036 5784 raft_consensus.cc:2466] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0b10743963c243339440eed8d461304c in term 0.
I20250625 07:57:23.580982 5852 leader_election.cc:304] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b10743963c243339440eed8d461304c, afcd783b188d46f1a5934c097532c907; no voters:
I20250625 07:57:23.581624 6122 raft_consensus.cc:2802] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:57:23.581892 6122 raft_consensus.cc:491] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:57:23.582218 6122 raft_consensus.cc:3058] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:23.586928 6122 raft_consensus.cc:513] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:23.588223 6122 leader_election.cc:290] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [CANDIDATE]: Term 1 election: Requested vote from peers ca9c4b589f6f45f89952373beea90665 (127.5.85.66:45235), afcd783b188d46f1a5934c097532c907 (127.5.85.68:39635)
I20250625 07:57:23.589182 6051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a851a0efcec84427a4d2362d000a8f58" candidate_uuid: "0b10743963c243339440eed8d461304c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "afcd783b188d46f1a5934c097532c907"
I20250625 07:57:23.589105 5784 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a851a0efcec84427a4d2362d000a8f58" candidate_uuid: "0b10743963c243339440eed8d461304c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca9c4b589f6f45f89952373beea90665"
I20250625 07:57:23.589589 6051 raft_consensus.cc:3058] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:23.589931 5784 raft_consensus.cc:3058] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:23.594880 5784 raft_consensus.cc:2466] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b10743963c243339440eed8d461304c in term 1.
I20250625 07:57:23.594996 6051 raft_consensus.cc:2466] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b10743963c243339440eed8d461304c in term 1.
I20250625 07:57:23.595907 5852 leader_election.cc:304] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b10743963c243339440eed8d461304c, afcd783b188d46f1a5934c097532c907; no voters:
I20250625 07:57:23.596529 6122 raft_consensus.cc:2802] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:57:23.596890 6122 raft_consensus.cc:695] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [term 1 LEADER]: Becoming Leader. State: Replica: 0b10743963c243339440eed8d461304c, State: Running, Role: LEADER
I20250625 07:57:23.597565 6122 consensus_queue.cc:237] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:23.605489 5507 catalog_manager.cc:5582] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c reported cstate change: term changed from 0 to 1, leader changed from <none> to 0b10743963c243339440eed8d461304c (127.5.85.67). New cstate: current_term: 1 leader_uuid: "0b10743963c243339440eed8d461304c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } health_report { overall_health: UNKNOWN } } }
I20250625 07:57:23.640275 5461 external_mini_cluster.cc:934] 4 TS(s) registered with all masters
I20250625 07:57:23.643990 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ca9c4b589f6f45f89952373beea90665 to finish bootstrapping
I20250625 07:57:23.657837 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0b10743963c243339440eed8d461304c to finish bootstrapping
I20250625 07:57:23.668077 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver afcd783b188d46f1a5934c097532c907 to finish bootstrapping
I20250625 07:57:23.678565 5461 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250625 07:57:23.681697 5461 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "a851a0efcec84427a4d2362d000a8f58"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "ca9c4b589f6f45f89952373beea90665"
rpc_addresses {
host: "127.5.85.66"
port: 45235
}
}
ts_infos {
permanent_uuid: "0b10743963c243339440eed8d461304c"
rpc_addresses {
host: "127.5.85.67"
port: 46403
}
}
ts_infos {
permanent_uuid: "afcd783b188d46f1a5934c097532c907"
rpc_addresses {
host: "127.5.85.68"
port: 39635
}
}
W20250625 07:57:23.798986 6097 tablet.cc:2378] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:57:23.978181 6122 consensus_queue.cc:1035] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:57:23.995569 6122 consensus_queue.cc:1035] T a851a0efcec84427a4d2362d000a8f58 P 0b10743963c243339440eed8d461304c [LEADER]: Connected to new peer: Peer: permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 07:57:23.996810 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 5833
W20250625 07:57:24.021948 5985 connection.cc:537] server connection from 127.5.85.67:40677 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 07:57:24.022228 5493 connection.cc:537] server connection from 127.5.85.67:41831 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 07:57:24.022405 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 5475
I20250625 07:57:24.049271 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:41951
--webserver_interface=127.5.85.126
--webserver_port=45151
--builtin_ntp_servers=127.5.85.84:34341
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:41951 with env {}
W20250625 07:57:24.358366 6135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:24.359001 6135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:24.359457 6135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:24.392010 6135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:57:24.392333 6135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:24.392593 6135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:57:24.392836 6135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:57:24.430173 6135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:34341
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:41951
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:41951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=45151
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:24.431532 6135 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:24.433184 6135 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:24.447513 6141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:24.621919 5696 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:41951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:41951: connect: Connection refused (error 111)
W20250625 07:57:25.005441 5829 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:41951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:41951: connect: Connection refused (error 111)
W20250625 07:57:25.022713 6096 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:41951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:41951: connect: Connection refused (error 111)
I20250625 07:57:25.495420 6150 raft_consensus.cc:491] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b10743963c243339440eed8d461304c)
I20250625 07:57:25.495992 6150 raft_consensus.cc:513] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
W20250625 07:57:25.502929 5719 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.85.67:46403: connect: Connection refused (error 111)
I20250625 07:57:25.506904 6150 leader_election.cc:290] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0b10743963c243339440eed8d461304c (127.5.85.67:46403), afcd783b188d46f1a5934c097532c907 (127.5.85.68:39635)
W20250625 07:57:25.511907 5719 leader_election.cc:336] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b10743963c243339440eed8d461304c (127.5.85.67:46403): Network error: Client connection negotiation failed: client connection to 127.5.85.67:46403: connect: Connection refused (error 111)
I20250625 07:57:25.513566 6051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a851a0efcec84427a4d2362d000a8f58" candidate_uuid: "ca9c4b589f6f45f89952373beea90665" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "afcd783b188d46f1a5934c097532c907" is_pre_election: true
I20250625 07:57:25.514146 6051 raft_consensus.cc:2466] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9c4b589f6f45f89952373beea90665 in term 1.
I20250625 07:57:25.514809 6152 raft_consensus.cc:491] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 0b10743963c243339440eed8d461304c)
I20250625 07:57:25.515496 5719 leader_election.cc:304] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: afcd783b188d46f1a5934c097532c907, ca9c4b589f6f45f89952373beea90665; no voters: 0b10743963c243339440eed8d461304c
I20250625 07:57:25.515554 6152 raft_consensus.cc:513] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:25.516607 6150 raft_consensus.cc:2802] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 07:57:25.516991 6150 raft_consensus.cc:491] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 0b10743963c243339440eed8d461304c)
I20250625 07:57:25.517355 6150 raft_consensus.cc:3058] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 1 FOLLOWER]: Advancing to term 2
I20250625 07:57:25.518677 6152 leader_election.cc:290] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ca9c4b589f6f45f89952373beea90665 (127.5.85.66:45235), 0b10743963c243339440eed8d461304c (127.5.85.67:46403)
W20250625 07:57:25.523196 5986 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.85.67:46403: connect: Connection refused (error 111)
I20250625 07:57:25.523991 6150 raft_consensus.cc:513] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
W20250625 07:57:25.527984 5986 leader_election.cc:336] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 0b10743963c243339440eed8d461304c (127.5.85.67:46403): Network error: Client connection negotiation failed: client connection to 127.5.85.67:46403: connect: Connection refused (error 111)
W20250625 07:57:25.530157 5719 leader_election.cc:336] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 0b10743963c243339440eed8d461304c (127.5.85.67:46403): Network error: Client connection negotiation failed: client connection to 127.5.85.67:46403: connect: Connection refused (error 111)
I20250625 07:57:25.530870 6150 leader_election.cc:290] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [CANDIDATE]: Term 2 election: Requested vote from peers 0b10743963c243339440eed8d461304c (127.5.85.67:46403), afcd783b188d46f1a5934c097532c907 (127.5.85.68:39635)
I20250625 07:57:25.531184 6051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a851a0efcec84427a4d2362d000a8f58" candidate_uuid: "ca9c4b589f6f45f89952373beea90665" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "afcd783b188d46f1a5934c097532c907"
I20250625 07:57:25.531754 6051 raft_consensus.cc:3058] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 1 FOLLOWER]: Advancing to term 2
I20250625 07:57:25.537496 6051 raft_consensus.cc:2466] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9c4b589f6f45f89952373beea90665 in term 2.
I20250625 07:57:25.538815 5719 leader_election.cc:304] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: afcd783b188d46f1a5934c097532c907, ca9c4b589f6f45f89952373beea90665; no voters: 0b10743963c243339440eed8d461304c
I20250625 07:57:25.542114 6150 raft_consensus.cc:2802] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 07:57:25.542768 5784 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a851a0efcec84427a4d2362d000a8f58" candidate_uuid: "afcd783b188d46f1a5934c097532c907" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "ca9c4b589f6f45f89952373beea90665" is_pre_election: true
I20250625 07:57:25.547247 6150 raft_consensus.cc:695] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 2 LEADER]: Becoming Leader. State: Replica: ca9c4b589f6f45f89952373beea90665, State: Running, Role: LEADER
I20250625 07:57:25.548796 6150 consensus_queue.cc:237] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:25.557041 5986 leader_election.cc:304] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: afcd783b188d46f1a5934c097532c907; no voters: 0b10743963c243339440eed8d461304c, ca9c4b589f6f45f89952373beea90665
I20250625 07:57:25.558231 6152 raft_consensus.cc:2747] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250625 07:57:24.447688 6142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:24.449317 6135 server_base.cc:1048] running on GCE node
W20250625 07:57:24.448675 6144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:25.578965 6135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:25.581575 6135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:25.582988 6135 hybrid_clock.cc:648] HybridClock initialized: now 1750838245582937 us; error 53 us; skew 500 ppm
I20250625 07:57:25.583868 6135 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:25.590152 6135 webserver.cc:469] Webserver started at http://127.5.85.126:45151/ using document root <none> and password file <none>
I20250625 07:57:25.591190 6135 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:25.591439 6135 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:25.599108 6135 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250625 07:57:25.604027 6164 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:25.605183 6135 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 07:57:25.605578 6135 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "4272391e062e4e14b01ab1a5b5edec15"
format_stamp: "Formatted at 2025-06-25 07:57:15 on dist-test-slave-8zlz"
I20250625 07:57:25.607645 6135 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:25.652144 6135 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:25.653663 6135 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:25.654140 6135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:25.723487 6135 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:41951
I20250625 07:57:25.723552 6216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:41951 every 8 connection(s)
I20250625 07:57:25.726456 6135 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 07:57:25.728111 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 6135
I20250625 07:57:25.728736 5461 kudu-admin-test.cc:735] Forcing unsafe config change on tserver ca9c4b589f6f45f89952373beea90665
I20250625 07:57:25.739701 6217 sys_catalog.cc:263] Verifying existing consensus state
I20250625 07:57:25.745077 6217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15: Bootstrap starting.
I20250625 07:57:25.782240 6217 log.cc:826] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:25.805673 6217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:57:25.806491 6217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15: Bootstrap complete.
I20250625 07:57:25.827504 6217 raft_consensus.cc:357] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } }
I20250625 07:57:25.829756 6217 raft_consensus.cc:738] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4272391e062e4e14b01ab1a5b5edec15, State: Initialized, Role: FOLLOWER
I20250625 07:57:25.830508 6217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } }
I20250625 07:57:25.831070 6217 raft_consensus.cc:397] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:57:25.831336 6217 raft_consensus.cc:491] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:57:25.831730 6217 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 1 FOLLOWER]: Advancing to term 2
I20250625 07:57:25.837388 6217 raft_consensus.cc:513] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } }
I20250625 07:57:25.838088 6217 leader_election.cc:304] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4272391e062e4e14b01ab1a5b5edec15; no voters:
I20250625 07:57:25.840058 6217 leader_election.cc:290] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 07:57:25.840458 6221 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 07:57:25.843562 6221 raft_consensus.cc:695] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [term 2 LEADER]: Becoming Leader. State: Replica: 4272391e062e4e14b01ab1a5b5edec15, State: Running, Role: LEADER
I20250625 07:57:25.844408 6221 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } }
I20250625 07:57:25.846158 6217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:57:25.854391 6222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4272391e062e4e14b01ab1a5b5edec15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } } }
I20250625 07:57:25.854425 6223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4272391e062e4e14b01ab1a5b5edec15. Latest consensus state: current_term: 2 leader_uuid: "4272391e062e4e14b01ab1a5b5edec15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4272391e062e4e14b01ab1a5b5edec15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 41951 } } }
I20250625 07:57:25.855404 6223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [sys.catalog]: This master's current role is: LEADER
I20250625 07:57:25.855403 6222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15 [sys.catalog]: This master's current role is: LEADER
I20250625 07:57:25.867027 6227 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:57:25.878434 6227 catalog_manager.cc:671] Loaded metadata for table TestTable [id=c2ee5f408cce44bea42accbe0d1470a1]
I20250625 07:57:25.885792 6227 tablet_loader.cc:96] loaded metadata for tablet a851a0efcec84427a4d2362d000a8f58 (table TestTable [id=c2ee5f408cce44bea42accbe0d1470a1])
I20250625 07:57:25.887851 6227 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:57:25.892545 6227 catalog_manager.cc:1261] Loaded cluster ID: c5cbada092964b60a648a84a2552d2dc
I20250625 07:57:25.892843 6227 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:57:25.901114 6227 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:57:25.906138 6227 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 4272391e062e4e14b01ab1a5b5edec15: Loaded TSK: 0
I20250625 07:57:25.907845 6227 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 07:57:26.040905 6096 heartbeater.cc:344] Connected to a master server at 127.5.85.126:41951
I20250625 07:57:26.047503 6182 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "afcd783b188d46f1a5934c097532c907" instance_seqno: 1750838243256571) as {username='slave'} at 127.5.85.68:44327; Asking this server to re-register.
I20250625 07:57:26.049331 6096 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:26.050001 6096 heartbeater.cc:507] Master 127.5.85.126:41951 requested a full tablet report, sending...
I20250625 07:57:26.053606 6182 ts_manager.cc:194] Registered new tserver with Master: afcd783b188d46f1a5934c097532c907 (127.5.85.68:39635)
W20250625 07:57:26.064402 6219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250625 07:57:26.064766 6051 raft_consensus.cc:1273] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 2 FOLLOWER]: Refusing update from remote peer ca9c4b589f6f45f89952373beea90665: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250625 07:57:26.065199 6219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250625 07:57:26.066241 6245 consensus_queue.cc:1035] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250625 07:57:26.075380 6182 catalog_manager.cc:5582] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 reported cstate change: term changed from 1 to 2, leader changed from 0b10743963c243339440eed8d461304c (127.5.85.67) to ca9c4b589f6f45f89952373beea90665 (127.5.85.66). New cstate: current_term: 2 leader_uuid: "ca9c4b589f6f45f89952373beea90665" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } }
W20250625 07:57:26.119172 6219 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 07:57:26.131960 5719 consensus_peers.cc:487] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 -> Peer 0b10743963c243339440eed8d461304c (127.5.85.67:46403): Couldn't send request to peer 0b10743963c243339440eed8d461304c. Status: Network error: Client connection negotiation failed: client connection to 127.5.85.67:46403: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250625 07:57:26.133849 5829 heartbeater.cc:344] Connected to a master server at 127.5.85.126:41951
I20250625 07:57:26.138258 6181 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "ca9c4b589f6f45f89952373beea90665" instance_seqno: 1750838239277302) as {username='slave'} at 127.5.85.66:34863; Asking this server to re-register.
I20250625 07:57:26.140336 5829 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:26.140964 5829 heartbeater.cc:507] Master 127.5.85.126:41951 requested a full tablet report, sending...
I20250625 07:57:26.147220 6182 ts_manager.cc:194] Registered new tserver with Master: ca9c4b589f6f45f89952373beea90665 (127.5.85.66:45235)
I20250625 07:57:26.673435 5696 heartbeater.cc:344] Connected to a master server at 127.5.85.126:41951
I20250625 07:57:26.675745 6182 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" instance_seqno: 1750838237498411) as {username='slave'} at 127.5.85.65:38153; Asking this server to re-register.
I20250625 07:57:26.677219 5696 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:26.677881 5696 heartbeater.cc:507] Master 127.5.85.126:41951 requested a full tablet report, sending...
I20250625 07:57:26.679973 6182 ts_manager.cc:194] Registered new tserver with Master: 726e287ed0c74cffb6e4812fb95c19f2 (127.5.85.65:36697)
W20250625 07:57:27.571979 6256 debug-util.cc:398] Leaking SignalData structure 0x7b0800036040 after lost signal to thread 6219
W20250625 07:57:27.572551 6256 kernel_stack_watchdog.cc:198] Thread 6219 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:641 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250625 07:57:27.636405 6219 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.468s user 0.558s sys 0.892s
W20250625 07:57:27.758092 6219 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.590s user 0.561s sys 0.898s
I20250625 07:57:27.805270 5784 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "ca9c4b589f6f45f89952373beea90665"
tablet_id: "a851a0efcec84427a4d2362d000a8f58"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "afcd783b188d46f1a5934c097532c907"
}
peers {
permanent_uuid: "ca9c4b589f6f45f89952373beea90665"
}
}
from {username='slave'} at 127.0.0.1:58100
W20250625 07:57:27.806663 5784 raft_consensus.cc:2216] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } unsafe_config_change: true
I20250625 07:57:27.807802 5784 raft_consensus.cc:3053] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 2 LEADER]: Stepping down as leader of term 2
I20250625 07:57:27.808109 5784 raft_consensus.cc:738] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: ca9c4b589f6f45f89952373beea90665, State: Running, Role: LEADER
I20250625 07:57:27.808799 5784 consensus_queue.cc:260] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:27.809979 5784 raft_consensus.cc:3058] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 2 FOLLOWER]: Advancing to term 3
I20250625 07:57:29.101227 6281 raft_consensus.cc:491] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ca9c4b589f6f45f89952373beea90665)
I20250625 07:57:29.101631 6281 raft_consensus.cc:513] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } }
I20250625 07:57:29.103180 6281 leader_election.cc:290] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ca9c4b589f6f45f89952373beea90665 (127.5.85.66:45235), 0b10743963c243339440eed8d461304c (127.5.85.67:46403)
I20250625 07:57:29.104409 5784 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a851a0efcec84427a4d2362d000a8f58" candidate_uuid: "afcd783b188d46f1a5934c097532c907" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "ca9c4b589f6f45f89952373beea90665" is_pre_election: true
W20250625 07:57:29.108788 5986 leader_election.cc:336] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 0b10743963c243339440eed8d461304c (127.5.85.67:46403): Network error: Client connection negotiation failed: client connection to 127.5.85.67:46403: connect: Connection refused (error 111)
I20250625 07:57:29.109095 5986 leader_election.cc:304] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: afcd783b188d46f1a5934c097532c907; no voters: 0b10743963c243339440eed8d461304c, ca9c4b589f6f45f89952373beea90665
I20250625 07:57:29.109624 6281 raft_consensus.cc:2747] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250625 07:57:29.384740 6284 raft_consensus.cc:491] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250625 07:57:29.385270 6284 raft_consensus.cc:513] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } unsafe_config_change: true
I20250625 07:57:29.386596 6284 leader_election.cc:290] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers afcd783b188d46f1a5934c097532c907 (127.5.85.68:39635)
I20250625 07:57:29.387714 6051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a851a0efcec84427a4d2362d000a8f58" candidate_uuid: "ca9c4b589f6f45f89952373beea90665" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "afcd783b188d46f1a5934c097532c907" is_pre_election: true
I20250625 07:57:29.388183 6051 raft_consensus.cc:2466] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ca9c4b589f6f45f89952373beea90665 in term 2.
I20250625 07:57:29.389151 5719 leader_election.cc:304] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: afcd783b188d46f1a5934c097532c907, ca9c4b589f6f45f89952373beea90665; no voters:
I20250625 07:57:29.389700 6284 raft_consensus.cc:2802] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250625 07:57:29.389994 6284 raft_consensus.cc:491] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250625 07:57:29.390249 6284 raft_consensus.cc:3058] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 3 FOLLOWER]: Advancing to term 4
I20250625 07:57:29.473217 6284 raft_consensus.cc:513] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } unsafe_config_change: true
I20250625 07:57:29.474311 6284 leader_election.cc:290] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [CANDIDATE]: Term 4 election: Requested vote from peers afcd783b188d46f1a5934c097532c907 (127.5.85.68:39635)
I20250625 07:57:29.475414 6051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a851a0efcec84427a4d2362d000a8f58" candidate_uuid: "ca9c4b589f6f45f89952373beea90665" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "afcd783b188d46f1a5934c097532c907"
I20250625 07:57:29.475811 6051 raft_consensus.cc:3058] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 2 FOLLOWER]: Advancing to term 4
I20250625 07:57:29.577447 6051 raft_consensus.cc:2466] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ca9c4b589f6f45f89952373beea90665 in term 4.
I20250625 07:57:29.578480 5719 leader_election.cc:304] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: afcd783b188d46f1a5934c097532c907, ca9c4b589f6f45f89952373beea90665; no voters:
I20250625 07:57:29.579147 6284 raft_consensus.cc:2802] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 4 FOLLOWER]: Leader election won for term 4
I20250625 07:57:29.580101 6284 raft_consensus.cc:695] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 4 LEADER]: Becoming Leader. State: Replica: ca9c4b589f6f45f89952373beea90665, State: Running, Role: LEADER
I20250625 07:57:29.580816 6284 consensus_queue.cc:237] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } unsafe_config_change: true
I20250625 07:57:29.587129 6182 catalog_manager.cc:5582] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER ca9c4b589f6f45f89952373beea90665 (127.5.85.66), VOTER afcd783b188d46f1a5934c097532c907 (127.5.85.68). New cstate: current_term: 4 leader_uuid: "ca9c4b589f6f45f89952373beea90665" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0b10743963c243339440eed8d461304c" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 46403 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } unsafe_config_change: true }
I20250625 07:57:30.068396 6051 raft_consensus.cc:1273] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 4 FOLLOWER]: Refusing update from remote peer ca9c4b589f6f45f89952373beea90665: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250625 07:57:30.069664 6284 consensus_queue.cc:1035] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250625 07:57:30.077297 6285 raft_consensus.cc:2953] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 0b10743963c243339440eed8d461304c (127.5.85.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } unsafe_config_change: true }
I20250625 07:57:30.078943 6051 raft_consensus.cc:2953] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 0b10743963c243339440eed8d461304c (127.5.85.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } unsafe_config_change: true }
I20250625 07:57:30.186586 6182 catalog_manager.cc:5582] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 reported cstate change: config changed from index -1 to 3, VOTER 0b10743963c243339440eed8d461304c (127.5.85.67) evicted, no longer has a pending config: VOTER ca9c4b589f6f45f89952373beea90665 (127.5.85.66), VOTER afcd783b188d46f1a5934c097532c907 (127.5.85.68). New cstate: current_term: 4 leader_uuid: "ca9c4b589f6f45f89952373beea90665" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } unsafe_config_change: true }
W20250625 07:57:30.196805 6182 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet a851a0efcec84427a4d2362d000a8f58 on TS 0b10743963c243339440eed8d461304c: Not found: failed to reset TS proxy: Could not find TS for UUID 0b10743963c243339440eed8d461304c
I20250625 07:57:30.214524 5784 consensus_queue.cc:237] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: true } } unsafe_config_change: true
I20250625 07:57:30.221077 6051 raft_consensus.cc:1273] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 4 FOLLOWER]: Refusing update from remote peer ca9c4b589f6f45f89952373beea90665: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250625 07:57:30.222224 6285 consensus_queue.cc:1035] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250625 07:57:30.228024 6284 raft_consensus.cc:2953] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 726e287ed0c74cffb6e4812fb95c19f2 (127.5.85.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: true } } unsafe_config_change: true }
I20250625 07:57:30.229542 6051 raft_consensus.cc:2953] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 726e287ed0c74cffb6e4812fb95c19f2 (127.5.85.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: true } } unsafe_config_change: true }
W20250625 07:57:30.230988 5719 consensus_peers.cc:487] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 -> Peer 726e287ed0c74cffb6e4812fb95c19f2 (127.5.85.65:36697): Couldn't send request to peer 726e287ed0c74cffb6e4812fb95c19f2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a851a0efcec84427a4d2362d000a8f58. This is attempt 1: this message will repeat every 5th retry.
W20250625 07:57:30.249521 6167 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet a851a0efcec84427a4d2362d000a8f58 on TS 0b10743963c243339440eed8d461304c failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0b10743963c243339440eed8d461304c
I20250625 07:57:30.281667 6168 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a851a0efcec84427a4d2362d000a8f58 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250625 07:57:30.284632 6182 catalog_manager.cc:5582] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 reported cstate change: config changed from index 3 to 5, NON_VOTER 726e287ed0c74cffb6e4812fb95c19f2 (127.5.85.65) added. New cstate: current_term: 4 leader_uuid: "ca9c4b589f6f45f89952373beea90665" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: true } } unsafe_config_change: true }
I20250625 07:57:30.683061 6303 ts_tablet_manager.cc:927] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: Initiating tablet copy from peer ca9c4b589f6f45f89952373beea90665 (127.5.85.66:45235)
I20250625 07:57:30.685544 6303 tablet_copy_client.cc:323] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: tablet copy: Beginning tablet copy session from remote peer at address 127.5.85.66:45235
I20250625 07:57:30.696834 5804 tablet_copy_service.cc:140] P ca9c4b589f6f45f89952373beea90665: Received BeginTabletCopySession request for tablet a851a0efcec84427a4d2362d000a8f58 from peer 726e287ed0c74cffb6e4812fb95c19f2 ({username='slave'} at 127.5.85.65:38043)
I20250625 07:57:30.697280 5804 tablet_copy_service.cc:161] P ca9c4b589f6f45f89952373beea90665: Beginning new tablet copy session on tablet a851a0efcec84427a4d2362d000a8f58 from peer 726e287ed0c74cffb6e4812fb95c19f2 at {username='slave'} at 127.5.85.65:38043: session id = 726e287ed0c74cffb6e4812fb95c19f2-a851a0efcec84427a4d2362d000a8f58
I20250625 07:57:30.702380 5804 tablet_copy_source_session.cc:215] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 07:57:30.707547 6303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a851a0efcec84427a4d2362d000a8f58. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:30.729050 6303 tablet_copy_client.cc:806] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: tablet copy: Starting download of 0 data blocks...
I20250625 07:57:30.729779 6303 tablet_copy_client.cc:670] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: tablet copy: Starting download of 1 WAL segments...
I20250625 07:57:30.733225 6303 tablet_copy_client.cc:538] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 07:57:30.741573 6303 tablet_bootstrap.cc:492] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: Bootstrap starting.
I20250625 07:57:30.757687 6303 log.cc:826] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:30.771576 6303 tablet_bootstrap.cc:492] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:57:30.772468 6303 tablet_bootstrap.cc:492] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: Bootstrap complete.
I20250625 07:57:30.773216 6303 ts_tablet_manager.cc:1397] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: Time spent bootstrapping tablet: real 0.032s user 0.031s sys 0.000s
I20250625 07:57:30.790480 6303 raft_consensus.cc:357] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: true } } unsafe_config_change: true
I20250625 07:57:30.791464 6303 raft_consensus.cc:738] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 726e287ed0c74cffb6e4812fb95c19f2, State: Initialized, Role: LEARNER
I20250625 07:57:30.792043 6303 consensus_queue.cc:260] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: true } } unsafe_config_change: true
I20250625 07:57:30.795351 6303 ts_tablet_manager.cc:1428] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2: Time spent starting tablet: real 0.022s user 0.023s sys 0.001s
I20250625 07:57:30.796962 5804 tablet_copy_service.cc:342] P ca9c4b589f6f45f89952373beea90665: Request end of tablet copy session 726e287ed0c74cffb6e4812fb95c19f2-a851a0efcec84427a4d2362d000a8f58 received from {username='slave'} at 127.5.85.65:38043
I20250625 07:57:30.797305 5804 tablet_copy_service.cc:434] P ca9c4b589f6f45f89952373beea90665: ending tablet copy session 726e287ed0c74cffb6e4812fb95c19f2-a851a0efcec84427a4d2362d000a8f58 on tablet a851a0efcec84427a4d2362d000a8f58 with peer 726e287ed0c74cffb6e4812fb95c19f2
I20250625 07:57:31.288403 5651 raft_consensus.cc:1215] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250625 07:57:31.416749 6167 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet a851a0efcec84427a4d2362d000a8f58 on TS 0b10743963c243339440eed8d461304c failed: Not found: failed to reset TS proxy: Could not find TS for UUID 0b10743963c243339440eed8d461304c
I20250625 07:57:31.872450 6313 raft_consensus.cc:1062] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665: attempting to promote NON_VOTER 726e287ed0c74cffb6e4812fb95c19f2 to VOTER
I20250625 07:57:31.873934 6313 consensus_queue.cc:237] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: false } } unsafe_config_change: true
I20250625 07:57:31.878330 5651 raft_consensus.cc:1273] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2 [term 4 LEARNER]: Refusing update from remote peer ca9c4b589f6f45f89952373beea90665: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250625 07:57:31.879441 6315 consensus_queue.cc:1035] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250625 07:57:31.879544 6051 raft_consensus.cc:1273] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 4 FOLLOWER]: Refusing update from remote peer ca9c4b589f6f45f89952373beea90665: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250625 07:57:31.880750 6315 consensus_queue.cc:1035] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250625 07:57:31.886178 6313 raft_consensus.cc:2953] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 726e287ed0c74cffb6e4812fb95c19f2 (127.5.85.65) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: false } } unsafe_config_change: true }
I20250625 07:57:31.887691 5651 raft_consensus.cc:2953] T a851a0efcec84427a4d2362d000a8f58 P 726e287ed0c74cffb6e4812fb95c19f2 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 726e287ed0c74cffb6e4812fb95c19f2 (127.5.85.65) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: false } } unsafe_config_change: true }
I20250625 07:57:31.888653 6051 raft_consensus.cc:2953] T a851a0efcec84427a4d2362d000a8f58 P afcd783b188d46f1a5934c097532c907 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 726e287ed0c74cffb6e4812fb95c19f2 (127.5.85.65) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: false } } unsafe_config_change: true }
I20250625 07:57:31.897413 6182 catalog_manager.cc:5582] T a851a0efcec84427a4d2362d000a8f58 P ca9c4b589f6f45f89952373beea90665 reported cstate change: config changed from index 5 to 6, 726e287ed0c74cffb6e4812fb95c19f2 (127.5.85.65) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "ca9c4b589f6f45f89952373beea90665" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "ca9c4b589f6f45f89952373beea90665" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45235 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "afcd783b188d46f1a5934c097532c907" member_type: VOTER last_known_addr { host: "127.5.85.68" port: 39635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36697 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250625 07:57:31.939946 5461 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250625 07:57:31.953344 5461 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "a851a0efcec84427a4d2362d000a8f58"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "ca9c4b589f6f45f89952373beea90665"
rpc_addresses {
host: "127.5.85.66"
port: 45235
}
}
ts_infos {
permanent_uuid: "afcd783b188d46f1a5934c097532c907"
rpc_addresses {
host: "127.5.85.68"
port: 39635
}
}
ts_infos {
permanent_uuid: "726e287ed0c74cffb6e4812fb95c19f2"
rpc_addresses {
host: "127.5.85.65"
port: 36697
}
}
I20250625 07:57:31.955521 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 5567
I20250625 07:57:31.976673 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 5700
I20250625 07:57:32.000198 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 5966
I20250625 07:57:32.021946 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 6135
2025-06-25T07:57:32Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19505 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250625 07:57:32.075562 5461 test_util.cc:276] Using random seed: -871171569
I20250625 07:57:32.081336 5461 ts_itest-base.cc:115] Starting cluster with:
I20250625 07:57:32.081516 5461 ts_itest-base.cc:116] --------------
I20250625 07:57:32.081632 5461 ts_itest-base.cc:117] 3 tablet servers
I20250625 07:57:32.081743 5461 ts_itest-base.cc:118] 3 replicas per TS
I20250625 07:57:32.081848 5461 ts_itest-base.cc:119] --------------
2025-06-25T07:57:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T07:57:32Z Disabled control of system clock
I20250625 07:57:32.115427 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:33601
--webserver_interface=127.5.85.126
--webserver_port=0
--builtin_ntp_servers=127.5.85.84:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:33601
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250625 07:57:32.419463 6333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:32.420025 6333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:32.420428 6333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:32.451992 6333 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:57:32.452281 6333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:32.452484 6333 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:57:32.452688 6333 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:57:32.489174 6333 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:33601
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:33601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:32.490464 6333 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:32.492175 6333 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:32.507812 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:32.507802 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:32.507802 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:32.509053 6333 server_base.cc:1048] running on GCE node
I20250625 07:57:33.609017 6333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:33.611573 6333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:33.612924 6333 hybrid_clock.cc:648] HybridClock initialized: now 1750838253612898 us; error 49 us; skew 500 ppm
I20250625 07:57:33.613699 6333 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:33.623139 6333 webserver.cc:469] Webserver started at http://127.5.85.126:41401/ using document root <none> and password file <none>
I20250625 07:57:33.624053 6333 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:33.624264 6333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:33.624711 6333 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:33.629031 6333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "47eadfeee47b40bf8a4f78a6e719d5e0"
format_stamp: "Formatted at 2025-06-25 07:57:33 on dist-test-slave-8zlz"
I20250625 07:57:33.630082 6333 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "47eadfeee47b40bf8a4f78a6e719d5e0"
format_stamp: "Formatted at 2025-06-25 07:57:33 on dist-test-slave-8zlz"
I20250625 07:57:33.636899 6333 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250625 07:57:33.642102 6350 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:33.643042 6333 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 07:57:33.643343 6333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "47eadfeee47b40bf8a4f78a6e719d5e0"
format_stamp: "Formatted at 2025-06-25 07:57:33 on dist-test-slave-8zlz"
I20250625 07:57:33.643653 6333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:33.692202 6333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:33.693655 6333 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:33.694089 6333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:33.758400 6333 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:33601
I20250625 07:57:33.758466 6401 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:33601 every 8 connection(s)
I20250625 07:57:33.760970 6333 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 07:57:33.764417 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 6333
I20250625 07:57:33.765060 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 07:57:33.766654 6402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:33.791311 6402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0: Bootstrap starting.
I20250625 07:57:33.796959 6402 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:33.798810 6402 log.cc:826] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:33.803597 6402 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0: No bootstrap required, opened a new log
I20250625 07:57:33.821664 6402 raft_consensus.cc:357] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47eadfeee47b40bf8a4f78a6e719d5e0" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33601 } }
I20250625 07:57:33.822417 6402 raft_consensus.cc:383] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:33.822734 6402 raft_consensus.cc:738] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47eadfeee47b40bf8a4f78a6e719d5e0, State: Initialized, Role: FOLLOWER
I20250625 07:57:33.823614 6402 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47eadfeee47b40bf8a4f78a6e719d5e0" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33601 } }
I20250625 07:57:33.824272 6402 raft_consensus.cc:397] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:57:33.824595 6402 raft_consensus.cc:491] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:57:33.824926 6402 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:33.829228 6402 raft_consensus.cc:513] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47eadfeee47b40bf8a4f78a6e719d5e0" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33601 } }
I20250625 07:57:33.830063 6402 leader_election.cc:304] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47eadfeee47b40bf8a4f78a6e719d5e0; no voters:
I20250625 07:57:33.831982 6402 leader_election.cc:290] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:57:33.832763 6407 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:57:33.835341 6407 raft_consensus.cc:695] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [term 1 LEADER]: Becoming Leader. State: Replica: 47eadfeee47b40bf8a4f78a6e719d5e0, State: Running, Role: LEADER
I20250625 07:57:33.836227 6402 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:57:33.836122 6407 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47eadfeee47b40bf8a4f78a6e719d5e0" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33601 } }
I20250625 07:57:33.845965 6409 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47eadfeee47b40bf8a4f78a6e719d5e0. Latest consensus state: current_term: 1 leader_uuid: "47eadfeee47b40bf8a4f78a6e719d5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47eadfeee47b40bf8a4f78a6e719d5e0" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33601 } } }
I20250625 07:57:33.847463 6409 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [sys.catalog]: This master's current role is: LEADER
I20250625 07:57:33.847227 6408 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47eadfeee47b40bf8a4f78a6e719d5e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47eadfeee47b40bf8a4f78a6e719d5e0" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33601 } } }
I20250625 07:57:33.847965 6408 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0 [sys.catalog]: This master's current role is: LEADER
I20250625 07:57:33.852655 6417 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:57:33.863266 6417 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:57:33.880386 6417 catalog_manager.cc:1349] Generated new cluster ID: c37a8d362eb84ef6ba7ddab133f86216
I20250625 07:57:33.880669 6417 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:57:33.894310 6417 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 07:57:33.895885 6417 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:57:33.909360 6417 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 47eadfeee47b40bf8a4f78a6e719d5e0: Generated new TSK 0
I20250625 07:57:33.910326 6417 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 07:57:33.925844 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:0
--local_ip_for_outbound_sockets=127.5.85.65
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:33601
--builtin_ntp_servers=127.5.85.84:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250625 07:57:34.223939 6426 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250625 07:57:34.224584 6426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:34.224869 6426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:34.225391 6426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:34.257485 6426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:34.258325 6426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:57:34.294497 6426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:33601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:34.295868 6426 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:34.297556 6426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:34.314826 6433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:34.315064 6432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:34.316586 6435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:35.745498 6434 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1428 milliseconds
I20250625 07:57:35.745600 6426 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:57:35.746822 6426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:35.749303 6426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:35.750748 6426 hybrid_clock.cc:648] HybridClock initialized: now 1750838255750685 us; error 75 us; skew 500 ppm
I20250625 07:57:35.751544 6426 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:35.757855 6426 webserver.cc:469] Webserver started at http://127.5.85.65:36549/ using document root <none> and password file <none>
I20250625 07:57:35.758829 6426 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:35.759027 6426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:35.759482 6426 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:35.763831 6426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9f0224defe384e958b311ef6dadc94d6"
format_stamp: "Formatted at 2025-06-25 07:57:35 on dist-test-slave-8zlz"
I20250625 07:57:35.764897 6426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9f0224defe384e958b311ef6dadc94d6"
format_stamp: "Formatted at 2025-06-25 07:57:35 on dist-test-slave-8zlz"
I20250625 07:57:35.771709 6426 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250625 07:57:35.777452 6442 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:35.778586 6426 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250625 07:57:35.778952 6426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9f0224defe384e958b311ef6dadc94d6"
format_stamp: "Formatted at 2025-06-25 07:57:35 on dist-test-slave-8zlz"
I20250625 07:57:35.779269 6426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:35.844048 6426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:35.845566 6426 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:35.845996 6426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:35.848469 6426 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:57:35.852430 6426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:57:35.852638 6426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:35.852885 6426 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:57:35.853039 6426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:35.979825 6426 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:39647
I20250625 07:57:35.979933 6554 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:39647 every 8 connection(s)
I20250625 07:57:35.982425 6426 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 07:57:35.986503 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 6426
I20250625 07:57:35.986994 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 07:57:35.993486 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.66:0
--local_ip_for_outbound_sockets=127.5.85.66
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:33601
--builtin_ntp_servers=127.5.85.84:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250625 07:57:36.003988 6555 heartbeater.cc:344] Connected to a master server at 127.5.85.126:33601
I20250625 07:57:36.004565 6555 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:36.005903 6555 heartbeater.cc:507] Master 127.5.85.126:33601 requested a full tablet report, sending...
I20250625 07:57:36.009236 6367 ts_manager.cc:194] Registered new tserver with Master: 9f0224defe384e958b311ef6dadc94d6 (127.5.85.65:39647)
I20250625 07:57:36.011427 6367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:43601
W20250625 07:57:36.290030 6559 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250625 07:57:36.290690 6559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:36.290951 6559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:36.291669 6559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:36.324156 6559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:36.325042 6559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.66
I20250625 07:57:36.360148 6559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:33601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.66
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:36.361414 6559 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:36.363085 6559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:36.380223 6565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:37.016057 6555 heartbeater.cc:499] Master 127.5.85.126:33601 was elected leader, sending a full tablet report...
W20250625 07:57:36.380290 6566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:36.380213 6568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:36.382603 6559 server_base.cc:1048] running on GCE node
I20250625 07:57:37.469743 6559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:37.471977 6559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:37.473356 6559 hybrid_clock.cc:648] HybridClock initialized: now 1750838257473312 us; error 61 us; skew 500 ppm
I20250625 07:57:37.474143 6559 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:37.480109 6559 webserver.cc:469] Webserver started at http://127.5.85.66:36311/ using document root <none> and password file <none>
I20250625 07:57:37.481024 6559 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:37.481242 6559 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:37.481698 6559 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:37.486235 6559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a7cebd9d7c3d470db5132c4c625db951"
format_stamp: "Formatted at 2025-06-25 07:57:37 on dist-test-slave-8zlz"
I20250625 07:57:37.487407 6559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a7cebd9d7c3d470db5132c4c625db951"
format_stamp: "Formatted at 2025-06-25 07:57:37 on dist-test-slave-8zlz"
I20250625 07:57:37.494305 6559 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250625 07:57:37.499897 6575 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:37.501026 6559 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 07:57:37.501344 6559 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a7cebd9d7c3d470db5132c4c625db951"
format_stamp: "Formatted at 2025-06-25 07:57:37 on dist-test-slave-8zlz"
I20250625 07:57:37.501691 6559 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:37.546257 6559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:37.547787 6559 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:37.548245 6559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:37.550820 6559 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:57:37.555608 6559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:57:37.555919 6559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250625 07:57:37.556257 6559 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:57:37.556444 6559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:37.682866 6559 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.66:43059
I20250625 07:57:37.682981 6687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.66:43059 every 8 connection(s)
I20250625 07:57:37.685487 6559 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 07:57:37.695660 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 6559
I20250625 07:57:37.696044 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 07:57:37.701967 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.67:0
--local_ip_for_outbound_sockets=127.5.85.67
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:33601
--builtin_ntp_servers=127.5.85.84:41917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250625 07:57:37.706668 6688 heartbeater.cc:344] Connected to a master server at 127.5.85.126:33601
I20250625 07:57:37.707212 6688 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:37.708539 6688 heartbeater.cc:507] Master 127.5.85.126:33601 requested a full tablet report, sending...
I20250625 07:57:37.711200 6367 ts_manager.cc:194] Registered new tserver with Master: a7cebd9d7c3d470db5132c4c625db951 (127.5.85.66:43059)
I20250625 07:57:37.712487 6367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.66:35223
W20250625 07:57:38.004248 6692 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250625 07:57:38.005542 6692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:38.005831 6692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:38.006328 6692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:38.039219 6692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:38.040087 6692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.67
I20250625 07:57:38.074725 6692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:33601
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.67
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:38.076112 6692 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:38.077836 6692 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:38.093261 6701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:38.716141 6688 heartbeater.cc:499] Master 127.5.85.126:33601 was elected leader, sending a full tablet report...
W20250625 07:57:38.093323 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:38.093310 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:39.189497 6700 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1094 milliseconds
I20250625 07:57:39.189606 6692 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:57:39.190778 6692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:39.193264 6692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:39.194708 6692 hybrid_clock.cc:648] HybridClock initialized: now 1750838259194647 us; error 75 us; skew 500 ppm
I20250625 07:57:39.195462 6692 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:39.204985 6692 webserver.cc:469] Webserver started at http://127.5.85.67:44937/ using document root <none> and password file <none>
I20250625 07:57:39.205894 6692 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:39.206115 6692 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:39.206524 6692 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:39.210889 6692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0a881a30adc9438aac596a5a7dd804cb"
format_stamp: "Formatted at 2025-06-25 07:57:39 on dist-test-slave-8zlz"
I20250625 07:57:39.212038 6692 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0a881a30adc9438aac596a5a7dd804cb"
format_stamp: "Formatted at 2025-06-25 07:57:39 on dist-test-slave-8zlz"
I20250625 07:57:39.218834 6692 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250625 07:57:39.224649 6708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:39.225656 6692 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250625 07:57:39.225991 6692 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0a881a30adc9438aac596a5a7dd804cb"
format_stamp: "Formatted at 2025-06-25 07:57:39 on dist-test-slave-8zlz"
I20250625 07:57:39.226353 6692 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:39.278333 6692 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:39.279767 6692 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:39.280196 6692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:39.282577 6692 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:57:39.286543 6692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:57:39.286792 6692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250625 07:57:39.287035 6692 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:57:39.287189 6692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:39.412607 6692 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.67:41893
I20250625 07:57:39.412714 6820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.67:41893 every 8 connection(s)
I20250625 07:57:39.415130 6692 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 07:57:39.421335 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 6692
I20250625 07:57:39.421844 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 07:57:39.437625 6821 heartbeater.cc:344] Connected to a master server at 127.5.85.126:33601
I20250625 07:57:39.438047 6821 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:39.439190 6821 heartbeater.cc:507] Master 127.5.85.126:33601 requested a full tablet report, sending...
I20250625 07:57:39.441524 6367 ts_manager.cc:194] Registered new tserver with Master: 0a881a30adc9438aac596a5a7dd804cb (127.5.85.67:41893)
I20250625 07:57:39.442732 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:57:39.442826 6367 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.67:40473
I20250625 07:57:39.476639 6367 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58364:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250625 07:57:39.495168 6367 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:57:39.553376 6490 tablet_service.cc:1468] Processing CreateTablet for tablet 21aad00449244f74bcde397a3ca0b899 (DEFAULT_TABLE table=TestTable [id=1abeae418d3645d293ebe522b8aee57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:57:39.553679 6756 tablet_service.cc:1468] Processing CreateTablet for tablet 21aad00449244f74bcde397a3ca0b899 (DEFAULT_TABLE table=TestTable [id=1abeae418d3645d293ebe522b8aee57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:57:39.555569 6756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21aad00449244f74bcde397a3ca0b899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:39.555554 6490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21aad00449244f74bcde397a3ca0b899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:39.557240 6623 tablet_service.cc:1468] Processing CreateTablet for tablet 21aad00449244f74bcde397a3ca0b899 (DEFAULT_TABLE table=TestTable [id=1abeae418d3645d293ebe522b8aee57b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:57:39.561388 6623 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21aad00449244f74bcde397a3ca0b899. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:39.584319 6840 tablet_bootstrap.cc:492] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951: Bootstrap starting.
I20250625 07:57:39.585244 6841 tablet_bootstrap.cc:492] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6: Bootstrap starting.
I20250625 07:57:39.587059 6842 tablet_bootstrap.cc:492] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb: Bootstrap starting.
I20250625 07:57:39.591995 6840 tablet_bootstrap.cc:654] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:39.593776 6840 log.cc:826] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:39.594573 6842 tablet_bootstrap.cc:654] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:39.595197 6841 tablet_bootstrap.cc:654] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:39.597004 6842 log.cc:826] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:39.597683 6841 log.cc:826] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:39.598565 6840 tablet_bootstrap.cc:492] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951: No bootstrap required, opened a new log
I20250625 07:57:39.599261 6840 ts_tablet_manager.cc:1397] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250625 07:57:39.602403 6842 tablet_bootstrap.cc:492] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb: No bootstrap required, opened a new log
I20250625 07:57:39.602891 6842 ts_tablet_manager.cc:1397] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb: Time spent bootstrapping tablet: real 0.016s user 0.016s sys 0.000s
I20250625 07:57:39.603385 6841 tablet_bootstrap.cc:492] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6: No bootstrap required, opened a new log
I20250625 07:57:39.603928 6841 ts_tablet_manager.cc:1397] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20250625 07:57:39.620813 6842 raft_consensus.cc:357] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:39.621765 6842 raft_consensus.cc:738] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a881a30adc9438aac596a5a7dd804cb, State: Initialized, Role: FOLLOWER
I20250625 07:57:39.622507 6842 consensus_queue.cc:260] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:39.625694 6821 heartbeater.cc:499] Master 127.5.85.126:33601 was elected leader, sending a full tablet report...
I20250625 07:57:39.625571 6840 raft_consensus.cc:357] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:39.626469 6840 raft_consensus.cc:738] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7cebd9d7c3d470db5132c4c625db951, State: Initialized, Role: FOLLOWER
I20250625 07:57:39.626720 6842 ts_tablet_manager.cc:1428] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb: Time spent starting tablet: real 0.024s user 0.020s sys 0.004s
I20250625 07:57:39.627187 6840 consensus_queue.cc:260] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:39.630139 6841 raft_consensus.cc:357] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:39.631270 6840 ts_tablet_manager.cc:1428] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951: Time spent starting tablet: real 0.032s user 0.027s sys 0.004s
I20250625 07:57:39.631191 6841 raft_consensus.cc:738] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f0224defe384e958b311ef6dadc94d6, State: Initialized, Role: FOLLOWER
I20250625 07:57:39.631954 6841 consensus_queue.cc:260] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:39.635968 6841 ts_tablet_manager.cc:1428] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6: Time spent starting tablet: real 0.032s user 0.030s sys 0.000s
I20250625 07:57:39.654863 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:57:39.658160 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9f0224defe384e958b311ef6dadc94d6 to finish bootstrapping
W20250625 07:57:39.669381 6822 tablet.cc:2378] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:57:39.670192 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a7cebd9d7c3d470db5132c4c625db951 to finish bootstrapping
I20250625 07:57:39.681005 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0a881a30adc9438aac596a5a7dd804cb to finish bootstrapping
W20250625 07:57:39.691732 6689 tablet.cc:2378] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:57:39.727248 6510 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "21aad00449244f74bcde397a3ca0b899"
dest_uuid: "9f0224defe384e958b311ef6dadc94d6"
from {username='slave'} at 127.0.0.1:34950
I20250625 07:57:39.727793 6510 raft_consensus.cc:491] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250625 07:57:39.728060 6510 raft_consensus.cc:3058] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:39.732316 6510 raft_consensus.cc:513] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:39.734676 6510 leader_election.cc:290] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [CANDIDATE]: Term 1 election: Requested vote from peers a7cebd9d7c3d470db5132c4c625db951 (127.5.85.66:43059), 0a881a30adc9438aac596a5a7dd804cb (127.5.85.67:41893)
W20250625 07:57:39.740433 6556 tablet.cc:2378] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:57:39.746546 6643 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21aad00449244f74bcde397a3ca0b899" candidate_uuid: "9f0224defe384e958b311ef6dadc94d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "a7cebd9d7c3d470db5132c4c625db951"
I20250625 07:57:39.747089 5461 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250625 07:57:39.747344 6643 raft_consensus.cc:3058] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:39.748062 6776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21aad00449244f74bcde397a3ca0b899" candidate_uuid: "9f0224defe384e958b311ef6dadc94d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0a881a30adc9438aac596a5a7dd804cb"
I20250625 07:57:39.748790 6776 raft_consensus.cc:3058] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:39.754146 6643 raft_consensus.cc:2466] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f0224defe384e958b311ef6dadc94d6 in term 1.
I20250625 07:57:39.755638 6776 raft_consensus.cc:2466] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9f0224defe384e958b311ef6dadc94d6 in term 1.
I20250625 07:57:39.755786 6443 leader_election.cc:304] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f0224defe384e958b311ef6dadc94d6, a7cebd9d7c3d470db5132c4c625db951; no voters:
I20250625 07:57:39.757153 6848 raft_consensus.cc:2802] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:57:39.759532 6848 raft_consensus.cc:695] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 1 LEADER]: Becoming Leader. State: Replica: 9f0224defe384e958b311ef6dadc94d6, State: Running, Role: LEADER
I20250625 07:57:39.760433 6848 consensus_queue.cc:237] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:39.773221 6365 catalog_manager.cc:5582] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9f0224defe384e958b311ef6dadc94d6 (127.5.85.65). New cstate: current_term: 1 leader_uuid: "9f0224defe384e958b311ef6dadc94d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } health_report { overall_health: HEALTHY } } }
I20250625 07:57:39.853144 5461 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250625 07:57:40.058939 5461 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250625 07:57:40.219460 6848 consensus_queue.cc:1035] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:57:40.235174 6858 consensus_queue.cc:1035] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:57:41.941049 6510 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "21aad00449244f74bcde397a3ca0b899"
dest_uuid: "9f0224defe384e958b311ef6dadc94d6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34964
I20250625 07:57:41.941570 6510 raft_consensus.cc:604] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 1 LEADER]: Received request to transfer leadership
I20250625 07:57:42.199231 6874 raft_consensus.cc:991] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6: : Instructing follower a7cebd9d7c3d470db5132c4c625db951 to start an election
I20250625 07:57:42.199599 6885 raft_consensus.cc:1079] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 1 LEADER]: Signalling peer a7cebd9d7c3d470db5132c4c625db951 to start an election
I20250625 07:57:42.201064 6643 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "21aad00449244f74bcde397a3ca0b899"
dest_uuid: "a7cebd9d7c3d470db5132c4c625db951"
from {username='slave'} at 127.5.85.65:39091
I20250625 07:57:42.201558 6643 raft_consensus.cc:491] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250625 07:57:42.201794 6643 raft_consensus.cc:3058] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [term 1 FOLLOWER]: Advancing to term 2
I20250625 07:57:42.205634 6643 raft_consensus.cc:513] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:42.207773 6643 leader_election.cc:290] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [CANDIDATE]: Term 2 election: Requested vote from peers 0a881a30adc9438aac596a5a7dd804cb (127.5.85.67:41893), 9f0224defe384e958b311ef6dadc94d6 (127.5.85.65:39647)
I20250625 07:57:42.218451 6776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21aad00449244f74bcde397a3ca0b899" candidate_uuid: "a7cebd9d7c3d470db5132c4c625db951" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "0a881a30adc9438aac596a5a7dd804cb"
I20250625 07:57:42.219086 6776 raft_consensus.cc:3058] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb [term 1 FOLLOWER]: Advancing to term 2
I20250625 07:57:42.220165 6510 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "21aad00449244f74bcde397a3ca0b899" candidate_uuid: "a7cebd9d7c3d470db5132c4c625db951" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9f0224defe384e958b311ef6dadc94d6"
I20250625 07:57:42.220748 6510 raft_consensus.cc:3053] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 1 LEADER]: Stepping down as leader of term 1
I20250625 07:57:42.221053 6510 raft_consensus.cc:738] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9f0224defe384e958b311ef6dadc94d6, State: Running, Role: LEADER
I20250625 07:57:42.221666 6510 consensus_queue.cc:260] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:42.222903 6510 raft_consensus.cc:3058] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 1 FOLLOWER]: Advancing to term 2
I20250625 07:57:42.225018 6776 raft_consensus.cc:2466] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7cebd9d7c3d470db5132c4c625db951 in term 2.
I20250625 07:57:42.226051 6579 leader_election.cc:304] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a881a30adc9438aac596a5a7dd804cb, a7cebd9d7c3d470db5132c4c625db951; no voters:
I20250625 07:57:42.227864 6889 raft_consensus.cc:2802] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 07:57:42.228281 6510 raft_consensus.cc:2466] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7cebd9d7c3d470db5132c4c625db951 in term 2.
I20250625 07:57:42.229357 6889 raft_consensus.cc:695] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [term 2 LEADER]: Becoming Leader. State: Replica: a7cebd9d7c3d470db5132c4c625db951, State: Running, Role: LEADER
I20250625 07:57:42.230121 6889 consensus_queue.cc:237] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } }
I20250625 07:57:42.237428 6365 catalog_manager.cc:5582] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 reported cstate change: term changed from 1 to 2, leader changed from 9f0224defe384e958b311ef6dadc94d6 (127.5.85.65) to a7cebd9d7c3d470db5132c4c625db951 (127.5.85.66). New cstate: current_term: 2 leader_uuid: "a7cebd9d7c3d470db5132c4c625db951" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7cebd9d7c3d470db5132c4c625db951" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 43059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 } health_report { overall_health: UNKNOWN } } }
I20250625 07:57:42.768442 6776 raft_consensus.cc:1273] T 21aad00449244f74bcde397a3ca0b899 P 0a881a30adc9438aac596a5a7dd804cb [term 2 FOLLOWER]: Refusing update from remote peer a7cebd9d7c3d470db5132c4c625db951: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250625 07:57:42.769613 6898 consensus_queue.cc:1035] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0a881a30adc9438aac596a5a7dd804cb" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 41893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250625 07:57:42.782205 6510 raft_consensus.cc:1273] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 2 FOLLOWER]: Refusing update from remote peer a7cebd9d7c3d470db5132c4c625db951: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250625 07:57:42.789553 6901 consensus_queue.cc:1035] T 21aad00449244f74bcde397a3ca0b899 P a7cebd9d7c3d470db5132c4c625db951 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f0224defe384e958b311ef6dadc94d6" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250625 07:57:44.544692 6510 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "21aad00449244f74bcde397a3ca0b899"
dest_uuid: "9f0224defe384e958b311ef6dadc94d6"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:34966
I20250625 07:57:44.545589 6510 raft_consensus.cc:604] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 2 FOLLOWER]: Received request to transfer leadership
I20250625 07:57:44.546030 6510 raft_consensus.cc:612] T 21aad00449244f74bcde397a3ca0b899 P 9f0224defe384e958b311ef6dadc94d6 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250625 07:57:45.578397 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 6426
I20250625 07:57:45.604095 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 6559
I20250625 07:57:45.630859 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 6692
I20250625 07:57:45.652582 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 6333
2025-06-25T07:57:45Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (13626 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250625 07:57:45.701987 5461 test_util.cc:276] Using random seed: -857545141
I20250625 07:57:45.706102 5461 ts_itest-base.cc:115] Starting cluster with:
I20250625 07:57:45.706279 5461 ts_itest-base.cc:116] --------------
I20250625 07:57:45.706430 5461 ts_itest-base.cc:117] 3 tablet servers
I20250625 07:57:45.706559 5461 ts_itest-base.cc:118] 3 replicas per TS
I20250625 07:57:45.706766 5461 ts_itest-base.cc:119] --------------
2025-06-25T07:57:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T07:57:45Z Disabled control of system clock
I20250625 07:57:45.740391 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:45929
--webserver_interface=127.5.85.126
--webserver_port=0
--builtin_ntp_servers=127.5.85.84:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:45929 with env {}
W20250625 07:57:46.030421 6936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:46.031028 6936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:46.031437 6936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:46.062412 6936 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:57:46.062736 6936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:46.062943 6936 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:57:46.063135 6936 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:57:46.098126 6936 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:45929
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:45929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:46.099453 6936 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:46.101079 6936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:46.115878 6943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:46.116055 6942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:46.117264 6936 server_base.cc:1048] running on GCE node
W20250625 07:57:46.115912 6945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:47.214759 6936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:47.217273 6936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:47.218645 6936 hybrid_clock.cc:648] HybridClock initialized: now 1750838267218597 us; error 49 us; skew 500 ppm
I20250625 07:57:47.219436 6936 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:47.225728 6936 webserver.cc:469] Webserver started at http://127.5.85.126:36029/ using document root <none> and password file <none>
I20250625 07:57:47.226646 6936 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:47.226857 6936 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:47.227284 6936 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:47.231691 6936 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ee592780239b4d2f9f44d14797ae608d"
format_stamp: "Formatted at 2025-06-25 07:57:47 on dist-test-slave-8zlz"
I20250625 07:57:47.232753 6936 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ee592780239b4d2f9f44d14797ae608d"
format_stamp: "Formatted at 2025-06-25 07:57:47 on dist-test-slave-8zlz"
I20250625 07:57:47.239710 6936 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250625 07:57:47.245141 6952 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:47.246223 6936 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250625 07:57:47.246562 6936 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ee592780239b4d2f9f44d14797ae608d"
format_stamp: "Formatted at 2025-06-25 07:57:47 on dist-test-slave-8zlz"
I20250625 07:57:47.246935 6936 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:47.302081 6936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:47.303790 6936 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:47.304243 6936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:47.372255 6936 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:45929
I20250625 07:57:47.372362 7003 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:45929 every 8 connection(s)
I20250625 07:57:47.374992 6936 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 07:57:47.378779 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 6936
I20250625 07:57:47.379272 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 07:57:47.380890 7004 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:47.405788 7004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d: Bootstrap starting.
I20250625 07:57:47.411217 7004 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:47.412964 7004 log.cc:826] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:47.417272 7004 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d: No bootstrap required, opened a new log
I20250625 07:57:47.434154 7004 raft_consensus.cc:357] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee592780239b4d2f9f44d14797ae608d" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 45929 } }
I20250625 07:57:47.434900 7004 raft_consensus.cc:383] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:47.435142 7004 raft_consensus.cc:738] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee592780239b4d2f9f44d14797ae608d, State: Initialized, Role: FOLLOWER
I20250625 07:57:47.435838 7004 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee592780239b4d2f9f44d14797ae608d" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 45929 } }
I20250625 07:57:47.436344 7004 raft_consensus.cc:397] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:57:47.436614 7004 raft_consensus.cc:491] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:57:47.436893 7004 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:47.440943 7004 raft_consensus.cc:513] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee592780239b4d2f9f44d14797ae608d" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 45929 } }
I20250625 07:57:47.441676 7004 leader_election.cc:304] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee592780239b4d2f9f44d14797ae608d; no voters:
I20250625 07:57:47.443364 7004 leader_election.cc:290] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:57:47.444020 7009 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:57:47.445995 7009 raft_consensus.cc:695] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [term 1 LEADER]: Becoming Leader. State: Replica: ee592780239b4d2f9f44d14797ae608d, State: Running, Role: LEADER
I20250625 07:57:47.446760 7009 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee592780239b4d2f9f44d14797ae608d" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 45929 } }
I20250625 07:57:47.447753 7004 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:57:47.457387 7010 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee592780239b4d2f9f44d14797ae608d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee592780239b4d2f9f44d14797ae608d" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 45929 } } }
I20250625 07:57:47.457536 7011 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee592780239b4d2f9f44d14797ae608d. Latest consensus state: current_term: 1 leader_uuid: "ee592780239b4d2f9f44d14797ae608d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee592780239b4d2f9f44d14797ae608d" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 45929 } } }
I20250625 07:57:47.458302 7010 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [sys.catalog]: This master's current role is: LEADER
I20250625 07:57:47.458331 7011 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d [sys.catalog]: This master's current role is: LEADER
I20250625 07:57:47.465664 7018 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:57:47.475769 7018 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:57:47.492559 7018 catalog_manager.cc:1349] Generated new cluster ID: 7c45977a4141498293daa95a7ccdec68
I20250625 07:57:47.492815 7018 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:57:47.511806 7018 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 07:57:47.513229 7018 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:57:47.527956 7018 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ee592780239b4d2f9f44d14797ae608d: Generated new TSK 0
I20250625 07:57:47.528915 7018 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 07:57:47.551940 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:0
--local_ip_for_outbound_sockets=127.5.85.65
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:45929
--builtin_ntp_servers=127.5.85.84:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 07:57:47.864346 7028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:47.864851 7028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:47.865345 7028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:47.895850 7028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:47.896715 7028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:57:47.931944 7028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:45929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:47.933194 7028 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:47.934811 7028 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:47.952654 7034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:47.952706 7035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:47.953975 7037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:47.954943 7028 server_base.cc:1048] running on GCE node
I20250625 07:57:49.041266 7028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:49.043821 7028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:49.045284 7028 hybrid_clock.cc:648] HybridClock initialized: now 1750838269045229 us; error 74 us; skew 500 ppm
I20250625 07:57:49.046070 7028 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:49.052618 7028 webserver.cc:469] Webserver started at http://127.5.85.65:46361/ using document root <none> and password file <none>
I20250625 07:57:49.053530 7028 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:49.053746 7028 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:49.054198 7028 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:49.058916 7028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7cdb18f2ec854bf9a3cead1872cd98f3"
format_stamp: "Formatted at 2025-06-25 07:57:49 on dist-test-slave-8zlz"
I20250625 07:57:49.060046 7028 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7cdb18f2ec854bf9a3cead1872cd98f3"
format_stamp: "Formatted at 2025-06-25 07:57:49 on dist-test-slave-8zlz"
I20250625 07:57:49.067612 7028 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20250625 07:57:49.073448 7044 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:49.074563 7028 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250625 07:57:49.074913 7028 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7cdb18f2ec854bf9a3cead1872cd98f3"
format_stamp: "Formatted at 2025-06-25 07:57:49 on dist-test-slave-8zlz"
I20250625 07:57:49.075237 7028 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:49.123904 7028 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:49.125496 7028 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:49.126009 7028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:49.128571 7028 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:57:49.133106 7028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:57:49.133327 7028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:49.133551 7028 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:57:49.133702 7028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:49.273448 7028 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:36523
I20250625 07:57:49.273485 7156 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:36523 every 8 connection(s)
I20250625 07:57:49.276087 7028 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 07:57:49.282953 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 7028
I20250625 07:57:49.283370 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 07:57:49.289047 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.66:0
--local_ip_for_outbound_sockets=127.5.85.66
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:45929
--builtin_ntp_servers=127.5.85.84:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:57:49.296564 7157 heartbeater.cc:344] Connected to a master server at 127.5.85.126:45929
I20250625 07:57:49.296988 7157 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:49.297962 7157 heartbeater.cc:507] Master 127.5.85.126:45929 requested a full tablet report, sending...
I20250625 07:57:49.300462 6969 ts_manager.cc:194] Registered new tserver with Master: 7cdb18f2ec854bf9a3cead1872cd98f3 (127.5.85.65:36523)
I20250625 07:57:49.302351 6969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:53297
W20250625 07:57:49.586723 7161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:49.587218 7161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:49.587709 7161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:49.619203 7161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:49.620076 7161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.66
I20250625 07:57:49.655359 7161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:45929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.66
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:49.656690 7161 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:49.658337 7161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:49.673182 7168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:50.306344 7157 heartbeater.cc:499] Master 127.5.85.126:45929 was elected leader, sending a full tablet report...
W20250625 07:57:49.674487 7170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:49.673341 7167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:50.753186 7169 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:57:50.753234 7161 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:57:50.757503 7161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:50.760408 7161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:50.761976 7161 hybrid_clock.cc:648] HybridClock initialized: now 1750838270761942 us; error 58 us; skew 500 ppm
I20250625 07:57:50.762864 7161 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:50.769009 7161 webserver.cc:469] Webserver started at http://127.5.85.66:42867/ using document root <none> and password file <none>
I20250625 07:57:50.769918 7161 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:50.770115 7161 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:50.770538 7161 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:50.775100 7161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "0027a70706ab43a39fce501271fdead2"
format_stamp: "Formatted at 2025-06-25 07:57:50 on dist-test-slave-8zlz"
I20250625 07:57:50.776278 7161 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "0027a70706ab43a39fce501271fdead2"
format_stamp: "Formatted at 2025-06-25 07:57:50 on dist-test-slave-8zlz"
I20250625 07:57:50.783455 7161 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250625 07:57:50.789134 7177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:50.790221 7161 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250625 07:57:50.790536 7161 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "0027a70706ab43a39fce501271fdead2"
format_stamp: "Formatted at 2025-06-25 07:57:50 on dist-test-slave-8zlz"
I20250625 07:57:50.790971 7161 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:50.836900 7161 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:50.838408 7161 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:50.838896 7161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:50.841392 7161 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:57:50.845806 7161 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:57:50.846055 7161 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:50.846503 7161 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:57:50.846727 7161 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250625 07:57:50.975112 7161 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.66:34971
I20250625 07:57:50.975217 7289 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.66:34971 every 8 connection(s)
I20250625 07:57:50.977640 7161 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 07:57:50.980351 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 7161
I20250625 07:57:50.980870 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 07:57:50.988286 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.67:0
--local_ip_for_outbound_sockets=127.5.85.67
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:45929
--builtin_ntp_servers=127.5.85.84:40035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:57:51.000662 7290 heartbeater.cc:344] Connected to a master server at 127.5.85.126:45929
I20250625 07:57:51.001178 7290 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:51.002437 7290 heartbeater.cc:507] Master 127.5.85.126:45929 requested a full tablet report, sending...
I20250625 07:57:51.005081 6969 ts_manager.cc:194] Registered new tserver with Master: 0027a70706ab43a39fce501271fdead2 (127.5.85.66:34971)
I20250625 07:57:51.006878 6969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.66:47861
W20250625 07:57:51.282773 7294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:51.283257 7294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:57:51.283692 7294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:51.316044 7294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:57:51.316844 7294 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.67
I20250625 07:57:51.352128 7294 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:40035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:45929
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.67
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:57:51.353415 7294 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:57:51.355399 7294 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:57:51.371587 7300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:57:52.011008 7290 heartbeater.cc:499] Master 127.5.85.126:45929 was elected leader, sending a full tablet report...
W20250625 07:57:51.372334 7303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:51.372715 7301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:57:52.455397 7302 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:57:52.455463 7294 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:57:52.459564 7294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:57:52.461769 7294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:57:52.463140 7294 hybrid_clock.cc:648] HybridClock initialized: now 1750838272463079 us; error 54 us; skew 500 ppm
I20250625 07:57:52.463954 7294 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:57:52.470713 7294 webserver.cc:469] Webserver started at http://127.5.85.67:37177/ using document root <none> and password file <none>
I20250625 07:57:52.471673 7294 fs_manager.cc:362] Metadata directory not provided
I20250625 07:57:52.471887 7294 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:57:52.472383 7294 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:57:52.476926 7294 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "63c60d4bc3dc464483cc688802016d97"
format_stamp: "Formatted at 2025-06-25 07:57:52 on dist-test-slave-8zlz"
I20250625 07:57:52.478075 7294 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "63c60d4bc3dc464483cc688802016d97"
format_stamp: "Formatted at 2025-06-25 07:57:52 on dist-test-slave-8zlz"
I20250625 07:57:52.485257 7294 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250625 07:57:52.490849 7310 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:52.491886 7294 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20250625 07:57:52.492199 7294 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "63c60d4bc3dc464483cc688802016d97"
format_stamp: "Formatted at 2025-06-25 07:57:52 on dist-test-slave-8zlz"
I20250625 07:57:52.492516 7294 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:57:52.544544 7294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:57:52.545969 7294 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:57:52.546387 7294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:57:52.548858 7294 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:57:52.552886 7294 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:57:52.553084 7294 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:52.553341 7294 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:57:52.553493 7294 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:57:52.680766 7294 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.67:35165
I20250625 07:57:52.680882 7423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.67:35165 every 8 connection(s)
I20250625 07:57:52.683336 7294 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 07:57:52.691429 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 7294
I20250625 07:57:52.691985 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 07:57:52.703251 7424 heartbeater.cc:344] Connected to a master server at 127.5.85.126:45929
I20250625 07:57:52.703713 7424 heartbeater.cc:461] Registering TS with master...
I20250625 07:57:52.704882 7424 heartbeater.cc:507] Master 127.5.85.126:45929 requested a full tablet report, sending...
I20250625 07:57:52.706858 6969 ts_manager.cc:194] Registered new tserver with Master: 63c60d4bc3dc464483cc688802016d97 (127.5.85.67:35165)
I20250625 07:57:52.707980 6969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.67:43917
I20250625 07:57:52.711958 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:57:52.744287 6969 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33362:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250625 07:57:52.762603 6969 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:57:52.813447 7092 tablet_service.cc:1468] Processing CreateTablet for tablet 14acc2b42d5f4f2fb332792bda0f9c0d (DEFAULT_TABLE table=TestTable [id=d5ba52bd524c483a9c131e02673f99dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:57:52.815603 7092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14acc2b42d5f4f2fb332792bda0f9c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:52.835858 7225 tablet_service.cc:1468] Processing CreateTablet for tablet 14acc2b42d5f4f2fb332792bda0f9c0d (DEFAULT_TABLE table=TestTable [id=d5ba52bd524c483a9c131e02673f99dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:57:52.837565 7225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14acc2b42d5f4f2fb332792bda0f9c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:52.837237 7358 tablet_service.cc:1468] Processing CreateTablet for tablet 14acc2b42d5f4f2fb332792bda0f9c0d (DEFAULT_TABLE table=TestTable [id=d5ba52bd524c483a9c131e02673f99dd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:57:52.839435 7358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14acc2b42d5f4f2fb332792bda0f9c0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:52.842662 7443 tablet_bootstrap.cc:492] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3: Bootstrap starting.
I20250625 07:57:52.850330 7443 tablet_bootstrap.cc:654] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:52.852804 7443 log.cc:826] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:52.858559 7443 tablet_bootstrap.cc:492] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3: No bootstrap required, opened a new log
I20250625 07:57:52.859133 7443 ts_tablet_manager.cc:1397] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.011s
I20250625 07:57:52.864223 7445 tablet_bootstrap.cc:492] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2: Bootstrap starting.
I20250625 07:57:52.866349 7446 tablet_bootstrap.cc:492] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97: Bootstrap starting.
I20250625 07:57:52.872072 7445 tablet_bootstrap.cc:654] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:52.873592 7446 tablet_bootstrap.cc:654] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:52.873777 7445 log.cc:826] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:52.875835 7446 log.cc:826] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97: Log is configured to *not* fsync() on all Append() calls
I20250625 07:57:52.878957 7445 tablet_bootstrap.cc:492] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2: No bootstrap required, opened a new log
I20250625 07:57:52.879451 7445 ts_tablet_manager.cc:1397] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250625 07:57:52.881249 7446 tablet_bootstrap.cc:492] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97: No bootstrap required, opened a new log
I20250625 07:57:52.881727 7446 ts_tablet_manager.cc:1397] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20250625 07:57:52.884944 7443 raft_consensus.cc:357] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } }
I20250625 07:57:52.885875 7443 raft_consensus.cc:383] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:52.886164 7443 raft_consensus.cc:738] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cdb18f2ec854bf9a3cead1872cd98f3, State: Initialized, Role: FOLLOWER
I20250625 07:57:52.887053 7443 consensus_queue.cc:260] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } }
I20250625 07:57:52.891126 7443 ts_tablet_manager.cc:1428] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3: Time spent starting tablet: real 0.032s user 0.027s sys 0.005s
I20250625 07:57:52.902880 7445 raft_consensus.cc:357] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } }
I20250625 07:57:52.903818 7445 raft_consensus.cc:383] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:52.904124 7445 raft_consensus.cc:738] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0027a70706ab43a39fce501271fdead2, State: Initialized, Role: FOLLOWER
I20250625 07:57:52.905030 7445 consensus_queue.cc:260] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } }
I20250625 07:57:52.908008 7446 raft_consensus.cc:357] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } }
I20250625 07:57:52.908887 7446 raft_consensus.cc:383] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:52.908949 7445 ts_tablet_manager.cc:1428] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2: Time spent starting tablet: real 0.029s user 0.028s sys 0.000s
I20250625 07:57:52.909184 7446 raft_consensus.cc:738] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c60d4bc3dc464483cc688802016d97, State: Initialized, Role: FOLLOWER
I20250625 07:57:52.909893 7446 consensus_queue.cc:260] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } }
I20250625 07:57:52.912580 7424 heartbeater.cc:499] Master 127.5.85.126:45929 was elected leader, sending a full tablet report...
I20250625 07:57:52.913736 7446 ts_tablet_manager.cc:1428] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97: Time spent starting tablet: real 0.032s user 0.032s sys 0.000s
W20250625 07:57:52.937611 7425 tablet.cc:2378] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:57:52.974656 7450 raft_consensus.cc:491] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:57:52.975232 7450 raft_consensus.cc:513] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } }
I20250625 07:57:52.978416 7450 leader_election.cc:290] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cdb18f2ec854bf9a3cead1872cd98f3 (127.5.85.65:36523), 63c60d4bc3dc464483cc688802016d97 (127.5.85.67:35165)
W20250625 07:57:52.990337 7291 tablet.cc:2378] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:57:52.991369 7449 raft_consensus.cc:491] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:57:52.991911 7449 raft_consensus.cc:513] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } }
I20250625 07:57:52.993392 7112 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14acc2b42d5f4f2fb332792bda0f9c0d" candidate_uuid: "0027a70706ab43a39fce501271fdead2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" is_pre_election: true
I20250625 07:57:52.994184 7112 raft_consensus.cc:2466] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0027a70706ab43a39fce501271fdead2 in term 0.
I20250625 07:57:52.995121 7449 leader_election.cc:290] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 63c60d4bc3dc464483cc688802016d97 (127.5.85.67:35165), 0027a70706ab43a39fce501271fdead2 (127.5.85.66:34971)
I20250625 07:57:52.996167 7178 leader_election.cc:304] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0027a70706ab43a39fce501271fdead2, 7cdb18f2ec854bf9a3cead1872cd98f3; no voters:
I20250625 07:57:52.997059 7450 raft_consensus.cc:2802] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:57:52.997401 7450 raft_consensus.cc:491] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:57:52.997814 7450 raft_consensus.cc:3058] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:53.005224 7450 raft_consensus.cc:513] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } }
I20250625 07:57:53.009264 7450 leader_election.cc:290] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [CANDIDATE]: Term 1 election: Requested vote from peers 7cdb18f2ec854bf9a3cead1872cd98f3 (127.5.85.65:36523), 63c60d4bc3dc464483cc688802016d97 (127.5.85.67:35165)
I20250625 07:57:53.011162 7112 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14acc2b42d5f4f2fb332792bda0f9c0d" candidate_uuid: "0027a70706ab43a39fce501271fdead2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3"
I20250625 07:57:53.011754 7112 raft_consensus.cc:3058] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:53.019418 7378 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14acc2b42d5f4f2fb332792bda0f9c0d" candidate_uuid: "0027a70706ab43a39fce501271fdead2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c60d4bc3dc464483cc688802016d97" is_pre_election: true
I20250625 07:57:53.020399 7378 raft_consensus.cc:2466] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0027a70706ab43a39fce501271fdead2 in term 0.
I20250625 07:57:53.021831 7377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14acc2b42d5f4f2fb332792bda0f9c0d" candidate_uuid: "0027a70706ab43a39fce501271fdead2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c60d4bc3dc464483cc688802016d97"
I20250625 07:57:53.022466 7377 raft_consensus.cc:3058] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:53.022539 7245 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14acc2b42d5f4f2fb332792bda0f9c0d" candidate_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0027a70706ab43a39fce501271fdead2" is_pre_election: true
I20250625 07:57:53.023536 7245 raft_consensus.cc:2391] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7cdb18f2ec854bf9a3cead1872cd98f3 in current term 1: Already voted for candidate 0027a70706ab43a39fce501271fdead2 in this term.
I20250625 07:57:53.028545 7377 raft_consensus.cc:2466] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0027a70706ab43a39fce501271fdead2 in term 1.
I20250625 07:57:53.028616 7112 raft_consensus.cc:2466] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0027a70706ab43a39fce501271fdead2 in term 1.
I20250625 07:57:53.029579 7180 leader_election.cc:304] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0027a70706ab43a39fce501271fdead2, 63c60d4bc3dc464483cc688802016d97; no voters:
I20250625 07:57:53.030346 7450 raft_consensus.cc:2802] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:57:53.032277 7450 raft_consensus.cc:695] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [term 1 LEADER]: Becoming Leader. State: Replica: 0027a70706ab43a39fce501271fdead2, State: Running, Role: LEADER
I20250625 07:57:53.033150 7450 consensus_queue.cc:237] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } }
W20250625 07:57:53.033664 7158 tablet.cc:2378] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:57:53.033634 7377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14acc2b42d5f4f2fb332792bda0f9c0d" candidate_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c60d4bc3dc464483cc688802016d97" is_pre_election: true
I20250625 07:57:53.034402 7377 raft_consensus.cc:2391] T 14acc2b42d5f4f2fb332792bda0f9c0d P 63c60d4bc3dc464483cc688802016d97 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7cdb18f2ec854bf9a3cead1872cd98f3 in current term 1: Already voted for candidate 0027a70706ab43a39fce501271fdead2 in this term.
I20250625 07:57:53.035485 7047 leader_election.cc:304] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7cdb18f2ec854bf9a3cead1872cd98f3; no voters: 0027a70706ab43a39fce501271fdead2, 63c60d4bc3dc464483cc688802016d97
I20250625 07:57:53.036192 7449 raft_consensus.cc:2747] T 14acc2b42d5f4f2fb332792bda0f9c0d P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250625 07:57:53.044008 6967 catalog_manager.cc:5582] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0027a70706ab43a39fce501271fdead2 (127.5.85.66). New cstate: current_term: 1 leader_uuid: "0027a70706ab43a39fce501271fdead2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } health_report { overall_health: HEALTHY } } }
I20250625 07:57:53.088632 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:57:53.092123 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7cdb18f2ec854bf9a3cead1872cd98f3 to finish bootstrapping
I20250625 07:57:53.104765 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0027a70706ab43a39fce501271fdead2 to finish bootstrapping
I20250625 07:57:53.116113 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 63c60d4bc3dc464483cc688802016d97 to finish bootstrapping
I20250625 07:57:53.129742 6967 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33362:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250625 07:57:53.131283 6967 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:57:53.148015 7092 tablet_service.cc:1468] Processing CreateTablet for tablet c12a234e9e9c4e23a6461d13e5d57605 (DEFAULT_TABLE table=TestAnotherTable [id=35bfe04dfe714f0eb75517455beb2a26]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250625 07:57:53.148396 7225 tablet_service.cc:1468] Processing CreateTablet for tablet c12a234e9e9c4e23a6461d13e5d57605 (DEFAULT_TABLE table=TestAnotherTable [id=35bfe04dfe714f0eb75517455beb2a26]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250625 07:57:53.149155 7092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12a234e9e9c4e23a6461d13e5d57605. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:53.149185 7358 tablet_service.cc:1468] Processing CreateTablet for tablet c12a234e9e9c4e23a6461d13e5d57605 (DEFAULT_TABLE table=TestAnotherTable [id=35bfe04dfe714f0eb75517455beb2a26]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250625 07:57:53.149470 7225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12a234e9e9c4e23a6461d13e5d57605. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:53.150261 7358 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c12a234e9e9c4e23a6461d13e5d57605. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:57:53.161641 7445 tablet_bootstrap.cc:492] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2: Bootstrap starting.
I20250625 07:57:53.163705 7443 tablet_bootstrap.cc:492] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3: Bootstrap starting.
I20250625 07:57:53.164623 7446 tablet_bootstrap.cc:492] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97: Bootstrap starting.
I20250625 07:57:53.167835 7445 tablet_bootstrap.cc:654] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:53.169708 7443 tablet_bootstrap.cc:654] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:53.170713 7446 tablet_bootstrap.cc:654] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97: Neither blocks nor log segments found. Creating new log.
I20250625 07:57:53.173584 7445 tablet_bootstrap.cc:492] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2: No bootstrap required, opened a new log
I20250625 07:57:53.174024 7445 ts_tablet_manager.cc:1397] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20250625 07:57:53.176638 7445 raft_consensus.cc:357] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } }
I20250625 07:57:53.177328 7445 raft_consensus.cc:383] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:53.177593 7445 raft_consensus.cc:738] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0027a70706ab43a39fce501271fdead2, State: Initialized, Role: FOLLOWER
I20250625 07:57:53.178112 7445 consensus_queue.cc:260] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } }
I20250625 07:57:53.178975 7446 tablet_bootstrap.cc:492] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97: No bootstrap required, opened a new log
I20250625 07:57:53.179489 7446 ts_tablet_manager.cc:1397] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97: Time spent bootstrapping tablet: real 0.015s user 0.014s sys 0.000s
I20250625 07:57:53.180145 7445 ts_tablet_manager.cc:1428] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20250625 07:57:53.180857 7443 tablet_bootstrap.cc:492] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3: No bootstrap required, opened a new log
I20250625 07:57:53.181226 7443 ts_tablet_manager.cc:1397] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.012s
I20250625 07:57:53.182080 7446 raft_consensus.cc:357] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } }
I20250625 07:57:53.182583 7446 raft_consensus.cc:383] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:53.182865 7446 raft_consensus.cc:738] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c60d4bc3dc464483cc688802016d97, State: Initialized, Role: FOLLOWER
I20250625 07:57:53.183598 7446 consensus_queue.cc:260] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } }
I20250625 07:57:53.183822 7443 raft_consensus.cc:357] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } }
I20250625 07:57:53.184428 7443 raft_consensus.cc:383] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:57:53.184691 7443 raft_consensus.cc:738] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cdb18f2ec854bf9a3cead1872cd98f3, State: Initialized, Role: FOLLOWER
I20250625 07:57:53.185642 7446 ts_tablet_manager.cc:1428] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250625 07:57:53.185410 7443 consensus_queue.cc:260] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } }
I20250625 07:57:53.187484 7443 ts_tablet_manager.cc:1428] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3: Time spent starting tablet: real 0.006s user 0.004s sys 0.002s
I20250625 07:57:53.237526 7450 raft_consensus.cc:491] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:57:53.237911 7450 raft_consensus.cc:513] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } }
I20250625 07:57:53.239383 7450 leader_election.cc:290] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7cdb18f2ec854bf9a3cead1872cd98f3 (127.5.85.65:36523), 63c60d4bc3dc464483cc688802016d97 (127.5.85.67:35165)
I20250625 07:57:53.240130 7112 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c12a234e9e9c4e23a6461d13e5d57605" candidate_uuid: "0027a70706ab43a39fce501271fdead2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" is_pre_election: true
I20250625 07:57:53.240402 7377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c12a234e9e9c4e23a6461d13e5d57605" candidate_uuid: "0027a70706ab43a39fce501271fdead2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c60d4bc3dc464483cc688802016d97" is_pre_election: true
I20250625 07:57:53.240583 7112 raft_consensus.cc:2466] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0027a70706ab43a39fce501271fdead2 in term 0.
I20250625 07:57:53.240864 7377 raft_consensus.cc:2466] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0027a70706ab43a39fce501271fdead2 in term 0.
I20250625 07:57:53.241392 7178 leader_election.cc:304] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0027a70706ab43a39fce501271fdead2, 7cdb18f2ec854bf9a3cead1872cd98f3; no voters:
I20250625 07:57:53.242017 7450 raft_consensus.cc:2802] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:57:53.242257 7450 raft_consensus.cc:491] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:57:53.242486 7450 raft_consensus.cc:3058] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:53.246928 7450 raft_consensus.cc:513] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } }
I20250625 07:57:53.248230 7450 leader_election.cc:290] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [CANDIDATE]: Term 1 election: Requested vote from peers 7cdb18f2ec854bf9a3cead1872cd98f3 (127.5.85.65:36523), 63c60d4bc3dc464483cc688802016d97 (127.5.85.67:35165)
I20250625 07:57:53.248870 7112 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c12a234e9e9c4e23a6461d13e5d57605" candidate_uuid: "0027a70706ab43a39fce501271fdead2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3"
I20250625 07:57:53.249249 7377 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c12a234e9e9c4e23a6461d13e5d57605" candidate_uuid: "0027a70706ab43a39fce501271fdead2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "63c60d4bc3dc464483cc688802016d97"
I20250625 07:57:53.249404 7112 raft_consensus.cc:3058] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:53.249747 7377 raft_consensus.cc:3058] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:57:53.255506 7112 raft_consensus.cc:2466] T c12a234e9e9c4e23a6461d13e5d57605 P 7cdb18f2ec854bf9a3cead1872cd98f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0027a70706ab43a39fce501271fdead2 in term 1.
I20250625 07:57:53.255506 7377 raft_consensus.cc:2466] T c12a234e9e9c4e23a6461d13e5d57605 P 63c60d4bc3dc464483cc688802016d97 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0027a70706ab43a39fce501271fdead2 in term 1.
I20250625 07:57:53.256470 7178 leader_election.cc:304] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0027a70706ab43a39fce501271fdead2, 7cdb18f2ec854bf9a3cead1872cd98f3; no voters:
I20250625 07:57:53.257058 7450 raft_consensus.cc:2802] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:57:53.257380 7450 raft_consensus.cc:695] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [term 1 LEADER]: Becoming Leader. State: Replica: 0027a70706ab43a39fce501271fdead2, State: Running, Role: LEADER
I20250625 07:57:53.258026 7450 consensus_queue.cc:237] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } }
I20250625 07:57:53.264374 6969 catalog_manager.cc:5582] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0027a70706ab43a39fce501271fdead2 (127.5.85.66). New cstate: current_term: 1 leader_uuid: "0027a70706ab43a39fce501271fdead2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0027a70706ab43a39fce501271fdead2" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 34971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 } health_report { overall_health: UNKNOWN } } }
I20250625 07:57:53.475200 7450 consensus_queue.cc:1035] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:57:53.496291 7457 consensus_queue.cc:1035] T 14acc2b42d5f4f2fb332792bda0f9c0d P 0027a70706ab43a39fce501271fdead2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250625 07:57:53.599707 7464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:53.600303 7464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:53.631713 7464 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250625 07:57:53.751983 7468 consensus_queue.cc:1035] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7cdb18f2ec854bf9a3cead1872cd98f3" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 36523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 07:57:53.776932 7468 consensus_queue.cc:1035] T c12a234e9e9c4e23a6461d13e5d57605 P 0027a70706ab43a39fce501271fdead2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "63c60d4bc3dc464483cc688802016d97" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 35165 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 07:57:54.842658 7464 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.172s user 0.458s sys 0.711s
W20250625 07:57:54.843056 7464 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.173s user 0.461s sys 0.711s
W20250625 07:57:56.207540 7485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:56.208128 7485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:56.239225 7485 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 07:57:57.419672 7485 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.140s user 0.474s sys 0.665s
W20250625 07:57:57.420074 7485 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.140s user 0.474s sys 0.665s
W20250625 07:57:58.788938 7502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:57:58.789474 7502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:57:58.822845 7502 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 07:58:00.003203 7502 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.141s user 0.429s sys 0.711s
W20250625 07:58:00.003499 7502 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.141s user 0.429s sys 0.711s
W20250625 07:58:01.355628 7518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:01.356221 7518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:01.387621 7518 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 07:58:02.576014 7518 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.150s user 0.514s sys 0.634s
W20250625 07:58:02.576321 7518 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.150s user 0.514s sys 0.634s
I20250625 07:58:03.645570 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 7028
I20250625 07:58:03.668540 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 7161
I20250625 07:58:03.699247 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 7294
I20250625 07:58:03.719210 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 6936
2025-06-25T07:58:03Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18068 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250625 07:58:03.770865 5461 test_util.cc:276] Using random seed: -839476265
I20250625 07:58:03.774981 5461 ts_itest-base.cc:115] Starting cluster with:
I20250625 07:58:03.775148 5461 ts_itest-base.cc:116] --------------
I20250625 07:58:03.775266 5461 ts_itest-base.cc:117] 3 tablet servers
I20250625 07:58:03.775372 5461 ts_itest-base.cc:118] 3 replicas per TS
I20250625 07:58:03.775483 5461 ts_itest-base.cc:119] --------------
2025-06-25T07:58:03Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T07:58:03Z Disabled control of system clock
I20250625 07:58:03.808547 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:34447
--webserver_interface=127.5.85.126
--webserver_port=0
--builtin_ntp_servers=127.5.85.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:34447
--superuser_acl=no-such-user with env {}
W20250625 07:58:04.102064 7540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:04.102651 7540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:04.103097 7540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:04.134160 7540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:58:04.134447 7540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:04.134682 7540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:58:04.134881 7540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:58:04.170841 7540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:34447
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:34447
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:04.172128 7540 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:04.173769 7540 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:04.187705 7546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:04.188062 7547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:04.188177 7549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:05.289781 7548 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1099 milliseconds
I20250625 07:58:05.289902 7540 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:58:05.291245 7540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:05.293901 7540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:05.295390 7540 hybrid_clock.cc:648] HybridClock initialized: now 1750838285295334 us; error 56 us; skew 500 ppm
I20250625 07:58:05.296320 7540 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:05.303052 7540 webserver.cc:469] Webserver started at http://127.5.85.126:43481/ using document root <none> and password file <none>
I20250625 07:58:05.304081 7540 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:05.304318 7540 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:05.304817 7540 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:58:05.309412 7540 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "63c7805ca39c4af5a2bc50de67a24b15"
format_stamp: "Formatted at 2025-06-25 07:58:05 on dist-test-slave-8zlz"
I20250625 07:58:05.310660 7540 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "63c7805ca39c4af5a2bc50de67a24b15"
format_stamp: "Formatted at 2025-06-25 07:58:05 on dist-test-slave-8zlz"
I20250625 07:58:05.318265 7540 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250625 07:58:05.324261 7556 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:05.325497 7540 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 07:58:05.325896 7540 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "63c7805ca39c4af5a2bc50de67a24b15"
format_stamp: "Formatted at 2025-06-25 07:58:05 on dist-test-slave-8zlz"
I20250625 07:58:05.326270 7540 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:05.380864 7540 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:05.382385 7540 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:05.382866 7540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:05.451025 7540 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:34447
I20250625 07:58:05.451094 7607 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:34447 every 8 connection(s)
I20250625 07:58:05.453828 7540 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 07:58:05.459208 7608 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:05.463280 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 7540
I20250625 07:58:05.463675 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 07:58:05.482511 7608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15: Bootstrap starting.
I20250625 07:58:05.488973 7608 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15: Neither blocks nor log segments found. Creating new log.
I20250625 07:58:05.490784 7608 log.cc:826] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:05.495484 7608 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15: No bootstrap required, opened a new log
I20250625 07:58:05.513392 7608 raft_consensus.cc:357] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c7805ca39c4af5a2bc50de67a24b15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 34447 } }
I20250625 07:58:05.514361 7608 raft_consensus.cc:383] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:58:05.514679 7608 raft_consensus.cc:738] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 63c7805ca39c4af5a2bc50de67a24b15, State: Initialized, Role: FOLLOWER
I20250625 07:58:05.515596 7608 consensus_queue.cc:260] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c7805ca39c4af5a2bc50de67a24b15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 34447 } }
I20250625 07:58:05.516258 7608 raft_consensus.cc:397] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:05.516582 7608 raft_consensus.cc:491] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:05.516957 7608 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:58:05.520903 7608 raft_consensus.cc:513] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c7805ca39c4af5a2bc50de67a24b15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 34447 } }
I20250625 07:58:05.521577 7608 leader_election.cc:304] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 63c7805ca39c4af5a2bc50de67a24b15; no voters:
I20250625 07:58:05.523396 7608 leader_election.cc:290] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:58:05.524091 7613 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:58:05.526079 7613 raft_consensus.cc:695] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [term 1 LEADER]: Becoming Leader. State: Replica: 63c7805ca39c4af5a2bc50de67a24b15, State: Running, Role: LEADER
I20250625 07:58:05.526836 7613 consensus_queue.cc:237] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c7805ca39c4af5a2bc50de67a24b15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 34447 } }
I20250625 07:58:05.527418 7608 sys_catalog.cc:564] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:58:05.537114 7614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "63c7805ca39c4af5a2bc50de67a24b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c7805ca39c4af5a2bc50de67a24b15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 34447 } } }
I20250625 07:58:05.537854 7614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [sys.catalog]: This master's current role is: LEADER
I20250625 07:58:05.539453 7615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 63c7805ca39c4af5a2bc50de67a24b15. Latest consensus state: current_term: 1 leader_uuid: "63c7805ca39c4af5a2bc50de67a24b15" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "63c7805ca39c4af5a2bc50de67a24b15" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 34447 } } }
I20250625 07:58:05.540321 7615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15 [sys.catalog]: This master's current role is: LEADER
I20250625 07:58:05.542045 7623 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:58:05.553737 7623 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:58:05.571208 7623 catalog_manager.cc:1349] Generated new cluster ID: 8449df48a23a435aa82ce0c11e00a1c5
I20250625 07:58:05.571485 7623 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:58:05.599298 7623 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 07:58:05.600829 7623 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:58:05.616355 7623 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 63c7805ca39c4af5a2bc50de67a24b15: Generated new TSK 0
I20250625 07:58:05.617286 7623 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 07:58:05.626502 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:0
--local_ip_for_outbound_sockets=127.5.85.65
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:34447
--builtin_ntp_servers=127.5.85.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 07:58:05.929692 7632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:05.930204 7632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:05.930747 7632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:05.962571 7632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:05.963521 7632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:58:05.999776 7632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:34447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:06.001103 7632 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:06.002856 7632 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:06.019896 7639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:06.019935 7638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:06.023703 7632 server_base.cc:1048] running on GCE node
W20250625 07:58:06.021198 7641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:07.114028 7632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:07.116683 7632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:07.118167 7632 hybrid_clock.cc:648] HybridClock initialized: now 1750838287118102 us; error 78 us; skew 500 ppm
I20250625 07:58:07.119073 7632 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:07.126003 7632 webserver.cc:469] Webserver started at http://127.5.85.65:40645/ using document root <none> and password file <none>
I20250625 07:58:07.126971 7632 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:07.127171 7632 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:07.127620 7632 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:58:07.132313 7632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4ea4513aa10d4861ae24b5ea5c46aefd"
format_stamp: "Formatted at 2025-06-25 07:58:07 on dist-test-slave-8zlz"
I20250625 07:58:07.133435 7632 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4ea4513aa10d4861ae24b5ea5c46aefd"
format_stamp: "Formatted at 2025-06-25 07:58:07 on dist-test-slave-8zlz"
I20250625 07:58:07.140838 7632 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250625 07:58:07.146834 7648 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:07.147927 7632 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250625 07:58:07.148237 7632 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4ea4513aa10d4861ae24b5ea5c46aefd"
format_stamp: "Formatted at 2025-06-25 07:58:07 on dist-test-slave-8zlz"
I20250625 07:58:07.148556 7632 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:07.209115 7632 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:07.210675 7632 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:07.211143 7632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:07.213780 7632 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:07.218374 7632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:58:07.218581 7632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:07.218880 7632 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:58:07.219038 7632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:07.350538 7632 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:39769
I20250625 07:58:07.350672 7760 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:39769 every 8 connection(s)
I20250625 07:58:07.353035 7632 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 07:58:07.357095 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 7632
I20250625 07:58:07.357602 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 07:58:07.363946 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.66:0
--local_ip_for_outbound_sockets=127.5.85.66
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:34447
--builtin_ntp_servers=127.5.85.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:58:07.373564 7761 heartbeater.cc:344] Connected to a master server at 127.5.85.126:34447
I20250625 07:58:07.373986 7761 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:07.375021 7761 heartbeater.cc:507] Master 127.5.85.126:34447 requested a full tablet report, sending...
I20250625 07:58:07.377537 7573 ts_manager.cc:194] Registered new tserver with Master: 4ea4513aa10d4861ae24b5ea5c46aefd (127.5.85.65:39769)
I20250625 07:58:07.379482 7573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:49071
W20250625 07:58:07.664474 7765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:07.664999 7765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:07.665488 7765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:07.697113 7765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:07.697969 7765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.66
I20250625 07:58:07.732748 7765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:34447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.66
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:07.734064 7765 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:07.735764 7765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:07.751840 7774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:08.382771 7761 heartbeater.cc:499] Master 127.5.85.126:34447 was elected leader, sending a full tablet report...
W20250625 07:58:07.751894 7771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:07.751873 7772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:08.827530 7773 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:58:08.827641 7765 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:58:08.831323 7765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:08.833431 7765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:08.834733 7765 hybrid_clock.cc:648] HybridClock initialized: now 1750838288834691 us; error 56 us; skew 500 ppm
I20250625 07:58:08.835505 7765 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:08.845469 7765 webserver.cc:469] Webserver started at http://127.5.85.66:41545/ using document root <none> and password file <none>
I20250625 07:58:08.846359 7765 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:08.846547 7765 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:08.847010 7765 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:58:08.851409 7765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac"
format_stamp: "Formatted at 2025-06-25 07:58:08 on dist-test-slave-8zlz"
I20250625 07:58:08.852494 7765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac"
format_stamp: "Formatted at 2025-06-25 07:58:08 on dist-test-slave-8zlz"
I20250625 07:58:08.859205 7765 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250625 07:58:08.864463 7781 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:08.865437 7765 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250625 07:58:08.865731 7765 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac"
format_stamp: "Formatted at 2025-06-25 07:58:08 on dist-test-slave-8zlz"
I20250625 07:58:08.866042 7765 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:08.912279 7765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:08.913722 7765 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:08.914155 7765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:08.916602 7765 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:08.920773 7765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:58:08.920987 7765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:08.921241 7765 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:58:08.921402 7765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:09.048882 7765 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.66:36783
I20250625 07:58:09.049005 7893 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.66:36783 every 8 connection(s)
I20250625 07:58:09.051340 7765 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 07:58:09.053980 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 7765
I20250625 07:58:09.054523 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 07:58:09.060959 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.67:0
--local_ip_for_outbound_sockets=127.5.85.67
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:34447
--builtin_ntp_servers=127.5.85.84:42519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:58:09.072479 7894 heartbeater.cc:344] Connected to a master server at 127.5.85.126:34447
I20250625 07:58:09.072976 7894 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:09.074203 7894 heartbeater.cc:507] Master 127.5.85.126:34447 requested a full tablet report, sending...
I20250625 07:58:09.076608 7573 ts_manager.cc:194] Registered new tserver with Master: 7c063b5e799d4f0a9a9b0c1a4b2cbfac (127.5.85.66:36783)
I20250625 07:58:09.077798 7573 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.66:56791
W20250625 07:58:09.349926 7898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:09.350436 7898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:09.351179 7898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:09.382680 7898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:09.383543 7898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.67
I20250625 07:58:09.418751 7898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:42519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:34447
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.67
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:09.420022 7898 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:09.421629 7898 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:09.436535 7905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:10.081435 7894 heartbeater.cc:499] Master 127.5.85.126:34447 was elected leader, sending a full tablet report...
W20250625 07:58:09.437042 7904 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:09.438010 7907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:09.439133 7898 server_base.cc:1048] running on GCE node
I20250625 07:58:10.529302 7898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:10.531541 7898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:10.532918 7898 hybrid_clock.cc:648] HybridClock initialized: now 1750838290532884 us; error 50 us; skew 500 ppm
I20250625 07:58:10.533721 7898 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:10.540167 7898 webserver.cc:469] Webserver started at http://127.5.85.67:46779/ using document root <none> and password file <none>
I20250625 07:58:10.541548 7898 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:10.541882 7898 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:10.542395 7898 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:58:10.547084 7898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5034563eba3542279afb04580b362924"
format_stamp: "Formatted at 2025-06-25 07:58:10 on dist-test-slave-8zlz"
I20250625 07:58:10.548233 7898 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5034563eba3542279afb04580b362924"
format_stamp: "Formatted at 2025-06-25 07:58:10 on dist-test-slave-8zlz"
I20250625 07:58:10.555478 7898 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250625 07:58:10.561235 7914 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:10.562328 7898 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20250625 07:58:10.562701 7898 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5034563eba3542279afb04580b362924"
format_stamp: "Formatted at 2025-06-25 07:58:10 on dist-test-slave-8zlz"
I20250625 07:58:10.563053 7898 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:10.611368 7898 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:10.612838 7898 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:10.613265 7898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:10.615701 7898 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:10.619772 7898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:58:10.619987 7898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:10.620222 7898 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:58:10.620378 7898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:10.745877 7898 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.67:39755
I20250625 07:58:10.745978 8026 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.67:39755 every 8 connection(s)
I20250625 07:58:10.748468 7898 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 07:58:10.749706 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 7898
I20250625 07:58:10.750203 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 07:58:10.769111 8027 heartbeater.cc:344] Connected to a master server at 127.5.85.126:34447
I20250625 07:58:10.769580 8027 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:10.770699 8027 heartbeater.cc:507] Master 127.5.85.126:34447 requested a full tablet report, sending...
I20250625 07:58:10.772763 7572 ts_manager.cc:194] Registered new tserver with Master: 5034563eba3542279afb04580b362924 (127.5.85.67:39755)
I20250625 07:58:10.774129 7572 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.67:41407
I20250625 07:58:10.785193 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:58:10.814795 7572 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38506:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250625 07:58:10.833696 7572 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:58:10.889802 7696 tablet_service.cc:1468] Processing CreateTablet for tablet 9d12a52143124f68abf98cd3e0ac14bf (DEFAULT_TABLE table=TestTable [id=a74e4a3e4cb14d248544aeb07841afac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:58:10.890935 7829 tablet_service.cc:1468] Processing CreateTablet for tablet 9d12a52143124f68abf98cd3e0ac14bf (DEFAULT_TABLE table=TestTable [id=a74e4a3e4cb14d248544aeb07841afac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:58:10.892043 7696 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d12a52143124f68abf98cd3e0ac14bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:10.892481 7829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d12a52143124f68abf98cd3e0ac14bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:10.892328 7962 tablet_service.cc:1468] Processing CreateTablet for tablet 9d12a52143124f68abf98cd3e0ac14bf (DEFAULT_TABLE table=TestTable [id=a74e4a3e4cb14d248544aeb07841afac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:58:10.894124 7962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d12a52143124f68abf98cd3e0ac14bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:10.912987 8046 tablet_bootstrap.cc:492] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac: Bootstrap starting.
I20250625 07:58:10.918694 8047 tablet_bootstrap.cc:492] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd: Bootstrap starting.
I20250625 07:58:10.919940 8046 tablet_bootstrap.cc:654] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac: Neither blocks nor log segments found. Creating new log.
I20250625 07:58:10.922262 8046 log.cc:826] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:10.923547 8048 tablet_bootstrap.cc:492] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924: Bootstrap starting.
I20250625 07:58:10.926683 8047 tablet_bootstrap.cc:654] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd: Neither blocks nor log segments found. Creating new log.
I20250625 07:58:10.928043 8046 tablet_bootstrap.cc:492] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac: No bootstrap required, opened a new log
I20250625 07:58:10.928508 8047 log.cc:826] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:10.928561 8046 ts_tablet_manager.cc:1397] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20250625 07:58:10.930596 8048 tablet_bootstrap.cc:654] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924: Neither blocks nor log segments found. Creating new log.
I20250625 07:58:10.932893 8048 log.cc:826] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:10.933033 8047 tablet_bootstrap.cc:492] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd: No bootstrap required, opened a new log
I20250625 07:58:10.933429 8047 ts_tablet_manager.cc:1397] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20250625 07:58:10.938318 8048 tablet_bootstrap.cc:492] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924: No bootstrap required, opened a new log
I20250625 07:58:10.938874 8048 ts_tablet_manager.cc:1397] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20250625 07:58:10.952948 8047 raft_consensus.cc:357] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 36783 } } peers { permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 } } peers { permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 } }
I20250625 07:58:10.953944 8047 raft_consensus.cc:383] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:58:10.954248 8047 raft_consensus.cc:738] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea4513aa10d4861ae24b5ea5c46aefd, State: Initialized, Role: FOLLOWER
I20250625 07:58:10.955183 8047 consensus_queue.cc:260] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 36783 } } peers { permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 } } peers { permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 } }
I20250625 07:58:10.955025 8046 raft_consensus.cc:357] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 36783 } } peers { permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 } } peers { permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 } }
I20250625 07:58:10.955964 8046 raft_consensus.cc:383] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:58:10.956279 8046 raft_consensus.cc:738] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c063b5e799d4f0a9a9b0c1a4b2cbfac, State: Initialized, Role: FOLLOWER
I20250625 07:58:10.957201 8046 consensus_queue.cc:260] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 36783 } } peers { permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 } } peers { permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 } }
I20250625 07:58:10.960186 8047 ts_tablet_manager.cc:1428] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd: Time spent starting tablet: real 0.027s user 0.023s sys 0.004s
I20250625 07:58:10.963356 8046 ts_tablet_manager.cc:1428] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac: Time spent starting tablet: real 0.034s user 0.026s sys 0.008s
I20250625 07:58:10.965788 8048 raft_consensus.cc:357] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 36783 } } peers { permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 } } peers { permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 } }
I20250625 07:58:10.966466 8048 raft_consensus.cc:383] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:58:10.966792 8048 raft_consensus.cc:738] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5034563eba3542279afb04580b362924, State: Initialized, Role: FOLLOWER
I20250625 07:58:10.967774 8048 consensus_queue.cc:260] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 36783 } } peers { permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 } } peers { permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 } }
I20250625 07:58:10.973943 8027 heartbeater.cc:499] Master 127.5.85.126:34447 was elected leader, sending a full tablet report...
I20250625 07:58:10.974342 8048 ts_tablet_manager.cc:1428] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924: Time spent starting tablet: real 0.035s user 0.029s sys 0.005s
W20250625 07:58:11.003384 8028 tablet.cc:2378] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 07:58:11.058008 7895 tablet.cc:2378] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 07:58:11.110973 7762 tablet.cc:2378] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:58:11.269106 8053 raft_consensus.cc:491] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:58:11.269599 8053 raft_consensus.cc:513] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 36783 } } peers { permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 } } peers { permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 } }
I20250625 07:58:11.272038 8053 leader_election.cc:290] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ea4513aa10d4861ae24b5ea5c46aefd (127.5.85.65:39769), 5034563eba3542279afb04580b362924 (127.5.85.67:39755)
I20250625 07:58:11.282984 7982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d12a52143124f68abf98cd3e0ac14bf" candidate_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5034563eba3542279afb04580b362924" is_pre_election: true
I20250625 07:58:11.282996 7716 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d12a52143124f68abf98cd3e0ac14bf" candidate_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" is_pre_election: true
I20250625 07:58:11.283663 7982 raft_consensus.cc:2466] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c063b5e799d4f0a9a9b0c1a4b2cbfac in term 0.
I20250625 07:58:11.283663 7716 raft_consensus.cc:2466] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7c063b5e799d4f0a9a9b0c1a4b2cbfac in term 0.
I20250625 07:58:11.284797 7785 leader_election.cc:304] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5034563eba3542279afb04580b362924, 7c063b5e799d4f0a9a9b0c1a4b2cbfac; no voters:
I20250625 07:58:11.285522 8053 raft_consensus.cc:2802] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:58:11.285851 8053 raft_consensus.cc:491] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:58:11.286068 8053 raft_consensus.cc:3058] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:58:11.290362 8053 raft_consensus.cc:513] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 36783 } } peers { permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 } } peers { permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 } }
I20250625 07:58:11.291815 8053 leader_election.cc:290] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [CANDIDATE]: Term 1 election: Requested vote from peers 4ea4513aa10d4861ae24b5ea5c46aefd (127.5.85.65:39769), 5034563eba3542279afb04580b362924 (127.5.85.67:39755)
I20250625 07:58:11.292637 7716 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d12a52143124f68abf98cd3e0ac14bf" candidate_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd"
I20250625 07:58:11.292826 7982 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d12a52143124f68abf98cd3e0ac14bf" candidate_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5034563eba3542279afb04580b362924"
I20250625 07:58:11.293071 7716 raft_consensus.cc:3058] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:58:11.293256 7982 raft_consensus.cc:3058] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:58:11.297833 7716 raft_consensus.cc:2466] T 9d12a52143124f68abf98cd3e0ac14bf P 4ea4513aa10d4861ae24b5ea5c46aefd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c063b5e799d4f0a9a9b0c1a4b2cbfac in term 1.
I20250625 07:58:11.297910 7982 raft_consensus.cc:2466] T 9d12a52143124f68abf98cd3e0ac14bf P 5034563eba3542279afb04580b362924 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7c063b5e799d4f0a9a9b0c1a4b2cbfac in term 1.
I20250625 07:58:11.298911 7784 leader_election.cc:304] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea4513aa10d4861ae24b5ea5c46aefd, 7c063b5e799d4f0a9a9b0c1a4b2cbfac; no voters:
I20250625 07:58:11.299583 8053 raft_consensus.cc:2802] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:58:11.301085 8053 raft_consensus.cc:695] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [term 1 LEADER]: Becoming Leader. State: Replica: 7c063b5e799d4f0a9a9b0c1a4b2cbfac, State: Running, Role: LEADER
I20250625 07:58:11.301820 8053 consensus_queue.cc:237] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 36783 } } peers { permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 } } peers { permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 } }
I20250625 07:58:11.313508 7572 catalog_manager.cc:5582] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac reported cstate change: term changed from 0 to 1, leader changed from <none> to 7c063b5e799d4f0a9a9b0c1a4b2cbfac (127.5.85.66). New cstate: current_term: 1 leader_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7c063b5e799d4f0a9a9b0c1a4b2cbfac" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 36783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 } health_report { overall_health: UNKNOWN } } }
I20250625 07:58:11.362257 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:58:11.365742 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4ea4513aa10d4861ae24b5ea5c46aefd to finish bootstrapping
I20250625 07:58:11.378947 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7c063b5e799d4f0a9a9b0c1a4b2cbfac to finish bootstrapping
I20250625 07:58:11.389706 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5034563eba3542279afb04580b362924 to finish bootstrapping
I20250625 07:58:11.705361 8053 consensus_queue.cc:1035] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [LEADER]: Connected to new peer: Peer: permanent_uuid: "5034563eba3542279afb04580b362924" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39755 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:58:11.724226 8053 consensus_queue.cc:1035] T 9d12a52143124f68abf98cd3e0ac14bf P 7c063b5e799d4f0a9a9b0c1a4b2cbfac [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea4513aa10d4861ae24b5ea5c46aefd" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 07:58:12.913425 7572 server_base.cc:1130] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:38522
I20250625 07:58:13.943272 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 7632
I20250625 07:58:13.963187 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 7765
I20250625 07:58:13.983620 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 7898
I20250625 07:58:14.003046 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 7540
2025-06-25T07:58:14Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (10286 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250625 07:58:14.057608 5461 test_util.cc:276] Using random seed: -829189522
I20250625 07:58:14.061661 5461 ts_itest-base.cc:115] Starting cluster with:
I20250625 07:58:14.061825 5461 ts_itest-base.cc:116] --------------
I20250625 07:58:14.061939 5461 ts_itest-base.cc:117] 3 tablet servers
I20250625 07:58:14.062044 5461 ts_itest-base.cc:118] 3 replicas per TS
I20250625 07:58:14.062153 5461 ts_itest-base.cc:119] --------------
2025-06-25T07:58:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T07:58:14Z Disabled control of system clock
I20250625 07:58:14.096323 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:43881
--webserver_interface=127.5.85.126
--webserver_port=0
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:43881 with env {}
W20250625 07:58:14.384734 8094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:14.385361 8094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:14.385803 8094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:14.417536 8094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:58:14.417826 8094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:14.418032 8094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:58:14.418226 8094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:58:14.453459 8094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:43881
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:43881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:14.454735 8094 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:14.456321 8094 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:14.471652 8101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:14.471750 8100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:14.471957 8103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:15.584754 8102 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1111 milliseconds
I20250625 07:58:15.584890 8094 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:58:15.586165 8094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:15.588737 8094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:15.590111 8094 hybrid_clock.cc:648] HybridClock initialized: now 1750838295590087 us; error 44 us; skew 500 ppm
I20250625 07:58:15.591063 8094 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:15.621601 8094 webserver.cc:469] Webserver started at http://127.5.85.126:44849/ using document root <none> and password file <none>
I20250625 07:58:15.622541 8094 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:15.622802 8094 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:15.623286 8094 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:58:15.627671 8094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "faf56aa493c943eda6ba3b0c6587d216"
format_stamp: "Formatted at 2025-06-25 07:58:15 on dist-test-slave-8zlz"
I20250625 07:58:15.628791 8094 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "faf56aa493c943eda6ba3b0c6587d216"
format_stamp: "Formatted at 2025-06-25 07:58:15 on dist-test-slave-8zlz"
I20250625 07:58:15.636050 8094 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250625 07:58:15.641667 8110 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:15.642822 8094 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250625 07:58:15.643151 8094 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "faf56aa493c943eda6ba3b0c6587d216"
format_stamp: "Formatted at 2025-06-25 07:58:15 on dist-test-slave-8zlz"
I20250625 07:58:15.643489 8094 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:15.705276 8094 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:15.706823 8094 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:15.707265 8094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:15.786746 8094 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:43881
I20250625 07:58:15.786832 8161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:43881 every 8 connection(s)
I20250625 07:58:15.789319 8094 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 07:58:15.790027 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 8094
I20250625 07:58:15.790400 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 07:58:15.795619 8162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:15.816629 8162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Bootstrap starting.
I20250625 07:58:15.822571 8162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Neither blocks nor log segments found. Creating new log.
I20250625 07:58:15.824223 8162 log.cc:826] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:15.828562 8162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: No bootstrap required, opened a new log
I20250625 07:58:15.845947 8162 raft_consensus.cc:357] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:15.846693 8162 raft_consensus.cc:383] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:58:15.846983 8162 raft_consensus.cc:738] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf56aa493c943eda6ba3b0c6587d216, State: Initialized, Role: FOLLOWER
I20250625 07:58:15.847770 8162 consensus_queue.cc:260] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:15.848348 8162 raft_consensus.cc:397] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:15.848652 8162 raft_consensus.cc:491] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:15.848913 8162 raft_consensus.cc:3058] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:58:15.852746 8162 raft_consensus.cc:513] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:15.853399 8162 leader_election.cc:304] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faf56aa493c943eda6ba3b0c6587d216; no voters:
I20250625 07:58:15.854967 8162 leader_election.cc:290] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:58:15.855659 8167 raft_consensus.cc:2802] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:58:15.858003 8167 raft_consensus.cc:695] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 1 LEADER]: Becoming Leader. State: Replica: faf56aa493c943eda6ba3b0c6587d216, State: Running, Role: LEADER
I20250625 07:58:15.858891 8162 sys_catalog.cc:564] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:58:15.858790 8167 consensus_queue.cc:237] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:15.869681 8169 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faf56aa493c943eda6ba3b0c6587d216. Latest consensus state: current_term: 1 leader_uuid: "faf56aa493c943eda6ba3b0c6587d216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } } }
I20250625 07:58:15.870249 8169 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: This master's current role is: LEADER
I20250625 07:58:15.871111 8168 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "faf56aa493c943eda6ba3b0c6587d216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } } }
I20250625 07:58:15.871701 8168 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: This master's current role is: LEADER
I20250625 07:58:15.874017 8176 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:58:15.885186 8176 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:58:15.902251 8176 catalog_manager.cc:1349] Generated new cluster ID: 93851fe16d7f4765aefd25619c599ef8
I20250625 07:58:15.902545 8176 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:58:15.916599 8176 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 07:58:15.918023 8176 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:58:15.929744 8176 catalog_manager.cc:5955] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Generated new TSK 0
I20250625 07:58:15.930622 8176 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 07:58:15.950354 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:0
--local_ip_for_outbound_sockets=127.5.85.65
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:43881
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 07:58:16.244433 8186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:16.244931 8186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:16.245426 8186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:16.276427 8186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:16.277279 8186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:58:16.313371 8186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:43881
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:16.314759 8186 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:16.316437 8186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:16.333774 8193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:16.340381 8186 server_base.cc:1048] running on GCE node
W20250625 07:58:16.335144 8192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:16.334239 8195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:17.449558 8186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:17.452157 8186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:17.453605 8186 hybrid_clock.cc:648] HybridClock initialized: now 1750838297453545 us; error 76 us; skew 500 ppm
I20250625 07:58:17.454407 8186 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:17.461356 8186 webserver.cc:469] Webserver started at http://127.5.85.65:35295/ using document root <none> and password file <none>
I20250625 07:58:17.462251 8186 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:17.462466 8186 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:17.462924 8186 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:58:17.467576 8186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f9b866b17f3446a6bd5e380d224c6aeb"
format_stamp: "Formatted at 2025-06-25 07:58:17 on dist-test-slave-8zlz"
I20250625 07:58:17.468765 8186 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f9b866b17f3446a6bd5e380d224c6aeb"
format_stamp: "Formatted at 2025-06-25 07:58:17 on dist-test-slave-8zlz"
I20250625 07:58:17.476327 8186 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250625 07:58:17.482184 8202 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:17.483296 8186 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 07:58:17.483606 8186 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f9b866b17f3446a6bd5e380d224c6aeb"
format_stamp: "Formatted at 2025-06-25 07:58:17 on dist-test-slave-8zlz"
I20250625 07:58:17.483927 8186 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:17.536871 8186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:17.538341 8186 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:17.538795 8186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:17.541635 8186 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:17.545904 8186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:58:17.546111 8186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:17.546377 8186 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:58:17.546530 8186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:17.696254 8186 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:43657
I20250625 07:58:17.696358 8314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:43657 every 8 connection(s)
I20250625 07:58:17.698719 8186 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 07:58:17.700819 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 8186
I20250625 07:58:17.701305 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 07:58:17.707571 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.66:0
--local_ip_for_outbound_sockets=127.5.85.66
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:43881
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:58:17.720284 8315 heartbeater.cc:344] Connected to a master server at 127.5.85.126:43881
I20250625 07:58:17.720790 8315 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:17.721998 8315 heartbeater.cc:507] Master 127.5.85.126:43881 requested a full tablet report, sending...
I20250625 07:58:17.724747 8127 ts_manager.cc:194] Registered new tserver with Master: f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65:43657)
I20250625 07:58:17.726550 8127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:60905
W20250625 07:58:17.996369 8319 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:17.996876 8319 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:17.997342 8319 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:18.028177 8319 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:18.029042 8319 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.66
I20250625 07:58:18.066089 8319 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:43881
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.66
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:18.067370 8319 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:18.068969 8319 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:18.084522 8328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:18.730453 8315 heartbeater.cc:499] Master 127.5.85.126:43881 was elected leader, sending a full tablet report...
W20250625 07:58:18.084703 8326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:18.085996 8319 server_base.cc:1048] running on GCE node
W20250625 07:58:18.084529 8325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:19.150599 8319 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:19.152769 8319 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:19.154107 8319 hybrid_clock.cc:648] HybridClock initialized: now 1750838299154082 us; error 42 us; skew 500 ppm
I20250625 07:58:19.154950 8319 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:19.161058 8319 webserver.cc:469] Webserver started at http://127.5.85.66:34965/ using document root <none> and password file <none>
I20250625 07:58:19.161953 8319 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:19.162164 8319 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:19.162613 8319 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:58:19.167088 8319 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cc8ab07892f84f6f910e5ee36cab6300"
format_stamp: "Formatted at 2025-06-25 07:58:19 on dist-test-slave-8zlz"
I20250625 07:58:19.168376 8319 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cc8ab07892f84f6f910e5ee36cab6300"
format_stamp: "Formatted at 2025-06-25 07:58:19 on dist-test-slave-8zlz"
I20250625 07:58:19.175438 8319 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250625 07:58:19.181022 8335 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:19.182121 8319 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 07:58:19.182447 8319 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cc8ab07892f84f6f910e5ee36cab6300"
format_stamp: "Formatted at 2025-06-25 07:58:19 on dist-test-slave-8zlz"
I20250625 07:58:19.182832 8319 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:19.240409 8319 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:19.241869 8319 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:19.242300 8319 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:19.244828 8319 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:19.248764 8319 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:58:19.248970 8319 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:19.249194 8319 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:58:19.249351 8319 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:19.376610 8319 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.66:45573
I20250625 07:58:19.376705 8447 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.66:45573 every 8 connection(s)
I20250625 07:58:19.379099 8319 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 07:58:19.386296 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 8319
I20250625 07:58:19.386755 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 07:58:19.393026 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.67:0
--local_ip_for_outbound_sockets=127.5.85.67
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:43881
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:58:19.401616 8448 heartbeater.cc:344] Connected to a master server at 127.5.85.126:43881
I20250625 07:58:19.402352 8448 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:19.404081 8448 heartbeater.cc:507] Master 127.5.85.126:43881 requested a full tablet report, sending...
I20250625 07:58:19.406934 8127 ts_manager.cc:194] Registered new tserver with Master: cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573)
I20250625 07:58:19.408694 8127 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.66:54517
W20250625 07:58:19.700673 8452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:19.701334 8452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:19.702100 8452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:19.737880 8452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:19.738792 8452 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.67
I20250625 07:58:19.774340 8452 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:43881
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.67
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:19.775723 8452 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:19.777431 8452 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:19.793332 8458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:20.413048 8448 heartbeater.cc:499] Master 127.5.85.126:43881 was elected leader, sending a full tablet report...
W20250625 07:58:19.794368 8459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:20.970899 8461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:20.971081 8460 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1179 milliseconds
W20250625 07:58:20.971726 8452 thread.cc:640] OpenStack (cloud detector) Time spent creating pthread: real 1.181s user 0.393s sys 0.774s
W20250625 07:58:20.971974 8452 thread.cc:606] OpenStack (cloud detector) Time spent starting thread: real 1.182s user 0.393s sys 0.774s
I20250625 07:58:20.972175 8452 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:58:20.973172 8452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:20.975458 8452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:20.976827 8452 hybrid_clock.cc:648] HybridClock initialized: now 1750838300976772 us; error 62 us; skew 500 ppm
I20250625 07:58:20.977613 8452 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:20.989286 8452 webserver.cc:469] Webserver started at http://127.5.85.67:36077/ using document root <none> and password file <none>
I20250625 07:58:20.990206 8452 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:20.990424 8452 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:20.990888 8452 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:58:20.995290 8452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "92b13ef756c54976a43d1e79ab68a602"
format_stamp: "Formatted at 2025-06-25 07:58:20 on dist-test-slave-8zlz"
I20250625 07:58:20.996398 8452 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "92b13ef756c54976a43d1e79ab68a602"
format_stamp: "Formatted at 2025-06-25 07:58:20 on dist-test-slave-8zlz"
I20250625 07:58:21.004097 8452 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250625 07:58:21.010048 8468 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:21.011276 8452 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 07:58:21.011610 8452 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "92b13ef756c54976a43d1e79ab68a602"
format_stamp: "Formatted at 2025-06-25 07:58:20 on dist-test-slave-8zlz"
I20250625 07:58:21.011950 8452 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:21.082707 8452 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:21.084160 8452 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:21.084615 8452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:21.087080 8452 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:21.091162 8452 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:58:21.091373 8452 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:21.091617 8452 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:58:21.091771 8452 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:21.220831 8452 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.67:45551
I20250625 07:58:21.220965 8580 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.67:45551 every 8 connection(s)
I20250625 07:58:21.223471 8452 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 07:58:21.230710 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 8452
I20250625 07:58:21.231102 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 07:58:21.244805 8581 heartbeater.cc:344] Connected to a master server at 127.5.85.126:43881
I20250625 07:58:21.245304 8581 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:21.246367 8581 heartbeater.cc:507] Master 127.5.85.126:43881 requested a full tablet report, sending...
I20250625 07:58:21.248555 8126 ts_manager.cc:194] Registered new tserver with Master: 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551)
I20250625 07:58:21.249843 8126 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.67:52257
I20250625 07:58:21.250838 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:58:21.282217 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:58:21.282567 5461 test_util.cc:276] Using random seed: -821964556
I20250625 07:58:21.322683 8126 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54788:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:58:21.362507 8250 tablet_service.cc:1468] Processing CreateTablet for tablet 557364b8ffd8492f8f2a2a616a7c127e (DEFAULT_TABLE table=TestTable [id=8967fd0b39fb4b15aa5cefddde446fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:58:21.363976 8250 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557364b8ffd8492f8f2a2a616a7c127e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:21.383630 8601 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap starting.
I20250625 07:58:21.389638 8601 tablet_bootstrap.cc:654] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Neither blocks nor log segments found. Creating new log.
I20250625 07:58:21.391484 8601 log.cc:826] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:21.396940 8601 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: No bootstrap required, opened a new log
I20250625 07:58:21.397423 8601 ts_tablet_manager.cc:1397] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20250625 07:58:21.414330 8601 raft_consensus.cc:357] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } }
I20250625 07:58:21.414984 8601 raft_consensus.cc:383] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:58:21.415258 8601 raft_consensus.cc:738] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b866b17f3446a6bd5e380d224c6aeb, State: Initialized, Role: FOLLOWER
I20250625 07:58:21.415920 8601 consensus_queue.cc:260] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } }
I20250625 07:58:21.416543 8601 raft_consensus.cc:397] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:21.416815 8601 raft_consensus.cc:491] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:21.417086 8601 raft_consensus.cc:3058] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:58:21.422240 8601 raft_consensus.cc:513] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } }
I20250625 07:58:21.423048 8601 leader_election.cc:304] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b866b17f3446a6bd5e380d224c6aeb; no voters:
I20250625 07:58:21.424767 8601 leader_election.cc:290] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:58:21.425127 8603 raft_consensus.cc:2802] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:58:21.427359 8603 raft_consensus.cc:695] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 1 LEADER]: Becoming Leader. State: Replica: f9b866b17f3446a6bd5e380d224c6aeb, State: Running, Role: LEADER
I20250625 07:58:21.428465 8603 consensus_queue.cc:237] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } }
I20250625 07:58:21.428968 8601 ts_tablet_manager.cc:1428] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Time spent starting tablet: real 0.031s user 0.019s sys 0.014s
I20250625 07:58:21.440953 8126 catalog_manager.cc:5582] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb reported cstate change: term changed from 0 to 1, leader changed from <none> to f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65). New cstate: current_term: 1 leader_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } health_report { overall_health: HEALTHY } } }
I20250625 07:58:21.651883 5461 test_util.cc:276] Using random seed: -821595247
I20250625 07:58:21.673048 8118 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54790:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:58:21.701836 8516 tablet_service.cc:1468] Processing CreateTablet for tablet 5f98cc844fb24ece82373c7f157fa07c (DEFAULT_TABLE table=TestTable1 [id=0e2a59e8f1c04c32a4656e8888cc3947]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:58:21.703475 8516 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f98cc844fb24ece82373c7f157fa07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:21.723748 8622 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Bootstrap starting.
I20250625 07:58:21.730875 8622 tablet_bootstrap.cc:654] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Neither blocks nor log segments found. Creating new log.
I20250625 07:58:21.733109 8622 log.cc:826] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:21.738354 8622 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: No bootstrap required, opened a new log
I20250625 07:58:21.738871 8622 ts_tablet_manager.cc:1397] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20250625 07:58:21.756058 8622 raft_consensus.cc:357] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } }
I20250625 07:58:21.756656 8622 raft_consensus.cc:383] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:58:21.756879 8622 raft_consensus.cc:738] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b13ef756c54976a43d1e79ab68a602, State: Initialized, Role: FOLLOWER
I20250625 07:58:21.757586 8622 consensus_queue.cc:260] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } }
I20250625 07:58:21.758143 8622 raft_consensus.cc:397] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:21.758419 8622 raft_consensus.cc:491] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:21.758785 8622 raft_consensus.cc:3058] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:58:21.763350 8622 raft_consensus.cc:513] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } }
I20250625 07:58:21.764140 8622 leader_election.cc:304] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92b13ef756c54976a43d1e79ab68a602; no voters:
I20250625 07:58:21.765678 8622 leader_election.cc:290] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:58:21.766186 8624 raft_consensus.cc:2802] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:58:21.768114 8624 raft_consensus.cc:695] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 1 LEADER]: Becoming Leader. State: Replica: 92b13ef756c54976a43d1e79ab68a602, State: Running, Role: LEADER
I20250625 07:58:21.768718 8581 heartbeater.cc:499] Master 127.5.85.126:43881 was elected leader, sending a full tablet report...
I20250625 07:58:21.769034 8624 consensus_queue.cc:237] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } }
I20250625 07:58:21.769395 8622 ts_tablet_manager.cc:1428] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Time spent starting tablet: real 0.030s user 0.028s sys 0.001s
I20250625 07:58:21.777866 8118 catalog_manager.cc:5582] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67). New cstate: current_term: 1 leader_uuid: "92b13ef756c54976a43d1e79ab68a602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } health_report { overall_health: HEALTHY } } }
I20250625 07:58:21.926924 5461 test_util.cc:276] Using random seed: -821320209
I20250625 07:58:21.948854 8120 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54796:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 07:58:21.978339 8383 tablet_service.cc:1468] Processing CreateTablet for tablet 0534efd80026410aae53e615ef097408 (DEFAULT_TABLE table=TestTable2 [id=5984b52a3e1a4ecb8e0138ebd0ffb2b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:58:21.979801 8383 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0534efd80026410aae53e615ef097408. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:21.999138 8644 tablet_bootstrap.cc:492] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap starting.
I20250625 07:58:22.004645 8644 tablet_bootstrap.cc:654] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Neither blocks nor log segments found. Creating new log.
I20250625 07:58:22.006302 8644 log.cc:826] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:22.010454 8644 tablet_bootstrap.cc:492] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: No bootstrap required, opened a new log
I20250625 07:58:22.010908 8644 ts_tablet_manager.cc:1397] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.007s
I20250625 07:58:22.028326 8644 raft_consensus.cc:357] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:22.028980 8644 raft_consensus.cc:383] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:58:22.029184 8644 raft_consensus.cc:738] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8ab07892f84f6f910e5ee36cab6300, State: Initialized, Role: FOLLOWER
I20250625 07:58:22.029848 8644 consensus_queue.cc:260] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:22.030370 8644 raft_consensus.cc:397] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:22.030678 8644 raft_consensus.cc:491] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:22.030989 8644 raft_consensus.cc:3058] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:58:22.035132 8644 raft_consensus.cc:513] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:22.035910 8644 leader_election.cc:304] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc8ab07892f84f6f910e5ee36cab6300; no voters:
I20250625 07:58:22.037628 8644 leader_election.cc:290] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:58:22.037994 8646 raft_consensus.cc:2802] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:58:22.041106 8646 raft_consensus.cc:695] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 1 LEADER]: Becoming Leader. State: Replica: cc8ab07892f84f6f910e5ee36cab6300, State: Running, Role: LEADER
I20250625 07:58:22.041985 8644 ts_tablet_manager.cc:1428] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Time spent starting tablet: real 0.031s user 0.026s sys 0.005s
I20250625 07:58:22.042040 8646 consensus_queue.cc:237] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:22.053100 8120 catalog_manager.cc:5582] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66). New cstate: current_term: 1 leader_uuid: "cc8ab07892f84f6f910e5ee36cab6300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } health_report { overall_health: HEALTHY } } }
I20250625 07:58:22.261981 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 8094
W20250625 07:58:22.457049 8315 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:43881 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:43881: connect: Connection refused (error 111)
W20250625 07:58:22.792752 8581 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:43881 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:43881: connect: Connection refused (error 111)
W20250625 07:58:23.069312 8448 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:43881 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:43881: connect: Connection refused (error 111)
I20250625 07:58:26.899125 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 8186
I20250625 07:58:26.918519 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 8319
I20250625 07:58:26.939402 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 8452
I20250625 07:58:26.962255 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:43881
--webserver_interface=127.5.85.126
--webserver_port=44849
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:43881 with env {}
W20250625 07:58:27.255630 8723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:27.256194 8723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:27.256644 8723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:27.287858 8723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:58:27.288199 8723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:27.288447 8723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:58:27.288684 8723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:58:27.324870 8723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:43881
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:43881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=44849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:27.326228 8723 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:27.328027 8723 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:27.343806 8729 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:27.344630 8730 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:27.343919 8732 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:27.346207 8723 server_base.cc:1048] running on GCE node
I20250625 07:58:28.451126 8723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:28.453575 8723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:28.454941 8723 hybrid_clock.cc:648] HybridClock initialized: now 1750838308454895 us; error 53 us; skew 500 ppm
I20250625 07:58:28.455717 8723 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:28.471863 8723 webserver.cc:469] Webserver started at http://127.5.85.126:44849/ using document root <none> and password file <none>
I20250625 07:58:28.472843 8723 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:28.473070 8723 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:28.480964 8723 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250625 07:58:28.485495 8739 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:28.486555 8723 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250625 07:58:28.486940 8723 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "faf56aa493c943eda6ba3b0c6587d216"
format_stamp: "Formatted at 2025-06-25 07:58:15 on dist-test-slave-8zlz"
I20250625 07:58:28.488852 8723 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:28.539222 8723 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:28.540678 8723 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:28.541149 8723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:28.609380 8723 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:43881
I20250625 07:58:28.609452 8790 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:43881 every 8 connection(s)
I20250625 07:58:28.612210 8723 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 07:58:28.614888 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 8723
I20250625 07:58:28.616348 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:43657
--local_ip_for_outbound_sockets=127.5.85.65
--tserver_master_addrs=127.5.85.126:43881
--webserver_port=35295
--webserver_interface=127.5.85.65
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:58:28.622547 8791 sys_catalog.cc:263] Verifying existing consensus state
I20250625 07:58:28.627339 8791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Bootstrap starting.
I20250625 07:58:28.637315 8791 log.cc:826] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:28.684567 8791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:28.685387 8791 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Bootstrap complete.
I20250625 07:58:28.705886 8791 raft_consensus.cc:357] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:28.708120 8791 raft_consensus.cc:738] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf56aa493c943eda6ba3b0c6587d216, State: Initialized, Role: FOLLOWER
I20250625 07:58:28.708909 8791 consensus_queue.cc:260] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:28.709443 8791 raft_consensus.cc:397] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:28.709718 8791 raft_consensus.cc:491] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:28.710018 8791 raft_consensus.cc:3058] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 2 FOLLOWER]: Advancing to term 3
I20250625 07:58:28.715891 8791 raft_consensus.cc:513] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:28.716611 8791 leader_election.cc:304] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faf56aa493c943eda6ba3b0c6587d216; no voters:
I20250625 07:58:28.718557 8791 leader_election.cc:290] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250625 07:58:28.719085 8795 raft_consensus.cc:2802] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 3 FOLLOWER]: Leader election won for term 3
I20250625 07:58:28.722055 8795 raft_consensus.cc:695] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 3 LEADER]: Becoming Leader. State: Replica: faf56aa493c943eda6ba3b0c6587d216, State: Running, Role: LEADER
I20250625 07:58:28.722966 8795 consensus_queue.cc:237] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:28.723397 8791 sys_catalog.cc:564] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:58:28.735018 8796 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "faf56aa493c943eda6ba3b0c6587d216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } } }
I20250625 07:58:28.735841 8796 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: This master's current role is: LEADER
I20250625 07:58:28.737920 8797 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faf56aa493c943eda6ba3b0c6587d216. Latest consensus state: current_term: 3 leader_uuid: "faf56aa493c943eda6ba3b0c6587d216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } } }
I20250625 07:58:28.739132 8797 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: This master's current role is: LEADER
I20250625 07:58:28.745795 8803 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:58:28.757881 8803 catalog_manager.cc:671] Loaded metadata for table TestTable [id=1e2c1d19957e4c61887a9bdbd4433e8b]
I20250625 07:58:28.759618 8803 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=5984b52a3e1a4ecb8e0138ebd0ffb2b6]
I20250625 07:58:28.761216 8803 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=94994e4d87fd4e79ba35dd32c106ced8]
I20250625 07:58:28.768913 8803 tablet_loader.cc:96] loaded metadata for tablet 0534efd80026410aae53e615ef097408 (table TestTable2 [id=5984b52a3e1a4ecb8e0138ebd0ffb2b6])
I20250625 07:58:28.770231 8803 tablet_loader.cc:96] loaded metadata for tablet 557364b8ffd8492f8f2a2a616a7c127e (table TestTable [id=1e2c1d19957e4c61887a9bdbd4433e8b])
I20250625 07:58:28.771626 8803 tablet_loader.cc:96] loaded metadata for tablet 5f98cc844fb24ece82373c7f157fa07c (table TestTable1 [id=94994e4d87fd4e79ba35dd32c106ced8])
I20250625 07:58:28.773114 8803 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:58:28.778278 8803 catalog_manager.cc:1261] Loaded cluster ID: 93851fe16d7f4765aefd25619c599ef8
I20250625 07:58:28.778617 8803 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:58:28.786684 8803 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:58:28.792104 8803 catalog_manager.cc:5966] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Loaded TSK: 0
I20250625 07:58:28.793798 8803 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250625 07:58:28.947774 8793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:28.948292 8793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:28.948813 8793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:28.981595 8793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:28.982491 8793 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:58:29.018388 8793 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:43657
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=35295
--tserver_master_addrs=127.5.85.126:43881
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:29.019785 8793 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:29.021507 8793 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:29.039556 8819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:29.039891 8818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:29.041294 8821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:30.129413 8820 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:58:30.129578 8793 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:58:30.131121 8793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:30.133818 8793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:30.135353 8793 hybrid_clock.cc:648] HybridClock initialized: now 1750838310135318 us; error 33 us; skew 500 ppm
I20250625 07:58:30.136453 8793 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:30.144217 8793 webserver.cc:469] Webserver started at http://127.5.85.65:35295/ using document root <none> and password file <none>
I20250625 07:58:30.145512 8793 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:30.145828 8793 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:30.156802 8793 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.004s
I20250625 07:58:30.163031 8828 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:30.164324 8793 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250625 07:58:30.164768 8793 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f9b866b17f3446a6bd5e380d224c6aeb"
format_stamp: "Formatted at 2025-06-25 07:58:17 on dist-test-slave-8zlz"
I20250625 07:58:30.167678 8793 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:30.245404 8793 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:30.247434 8793 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:30.247992 8793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:30.251279 8793 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:30.258729 8835 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 07:58:30.266424 8793 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 07:58:30.266783 8793 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250625 07:58:30.267117 8793 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 07:58:30.274310 8793 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 07:58:30.274592 8793 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.004s sys 0.000s
I20250625 07:58:30.275017 8835 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap starting.
I20250625 07:58:30.345783 8835 log.cc:826] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:30.462241 8793 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:43657
I20250625 07:58:30.462416 8942 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:43657 every 8 connection(s)
I20250625 07:58:30.465785 8793 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 07:58:30.468664 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 8793
I20250625 07:58:30.470362 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.66:45573
--local_ip_for_outbound_sockets=127.5.85.66
--tserver_master_addrs=127.5.85.126:43881
--webserver_port=34965
--webserver_interface=127.5.85.66
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:58:30.472419 8835 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:30.473387 8835 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap complete.
I20250625 07:58:30.475118 8835 ts_tablet_manager.cc:1397] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Time spent bootstrapping tablet: real 0.201s user 0.128s sys 0.064s
I20250625 07:58:30.497223 8835 raft_consensus.cc:357] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } }
I20250625 07:58:30.500126 8835 raft_consensus.cc:738] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b866b17f3446a6bd5e380d224c6aeb, State: Initialized, Role: FOLLOWER
I20250625 07:58:30.501101 8835 consensus_queue.cc:260] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } }
I20250625 07:58:30.501722 8835 raft_consensus.cc:397] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:30.502066 8835 raft_consensus.cc:491] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:30.502427 8835 raft_consensus.cc:3058] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 1 FOLLOWER]: Advancing to term 2
I20250625 07:58:30.510457 8835 raft_consensus.cc:513] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } }
I20250625 07:58:30.511389 8835 leader_election.cc:304] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9b866b17f3446a6bd5e380d224c6aeb; no voters:
I20250625 07:58:30.516409 8835 leader_election.cc:290] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 07:58:30.517330 8943 heartbeater.cc:344] Connected to a master server at 127.5.85.126:43881
I20250625 07:58:30.517829 8943 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:30.519143 8943 heartbeater.cc:507] Master 127.5.85.126:43881 requested a full tablet report, sending...
I20250625 07:58:30.520148 8948 raft_consensus.cc:2802] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Leader election won for term 2
I20250625 07:58:30.525276 8948 raft_consensus.cc:695] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 LEADER]: Becoming Leader. State: Replica: f9b866b17f3446a6bd5e380d224c6aeb, State: Running, Role: LEADER
I20250625 07:58:30.526144 8948 consensus_queue.cc:237] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } }
I20250625 07:58:30.530589 8835 ts_tablet_manager.cc:1428] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Time spent starting tablet: real 0.055s user 0.041s sys 0.008s
I20250625 07:58:30.541600 8756 ts_manager.cc:194] Registered new tserver with Master: f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65:43657)
I20250625 07:58:30.545764 8756 catalog_manager.cc:5582] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb reported cstate change: term changed from 0 to 2, leader changed from <none> to f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65), VOTER f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65) added. New cstate: current_term: 2 leader_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } health_report { overall_health: HEALTHY } } }
W20250625 07:58:30.595816 8756 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 557364b8ffd8492f8f2a2a616a7c127e with cas_config_opid_index -1: no extra replica candidate found for tablet 557364b8ffd8492f8f2a2a616a7c127e (table TestTable [id=1e2c1d19957e4c61887a9bdbd4433e8b]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
I20250625 07:58:30.598351 8756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:45699
I20250625 07:58:30.603241 8943 heartbeater.cc:499] Master 127.5.85.126:43881 was elected leader, sending a full tablet report...
W20250625 07:58:30.838426 8947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:30.838951 8947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:30.839414 8947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:30.871064 8947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:30.871866 8947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.66
I20250625 07:58:30.906021 8947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.66:45573
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.85.66
--webserver_port=34965
--tserver_master_addrs=127.5.85.126:43881
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.66
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:30.907297 8947 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:30.908866 8947 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:30.924165 8963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:30.924223 8965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:30.925585 8962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:30.926522 8947 server_base.cc:1048] running on GCE node
I20250625 07:58:32.001078 8947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:32.003329 8947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:32.004706 8947 hybrid_clock.cc:648] HybridClock initialized: now 1750838312004676 us; error 44 us; skew 500 ppm
I20250625 07:58:32.005508 8947 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:32.011929 8947 webserver.cc:469] Webserver started at http://127.5.85.66:34965/ using document root <none> and password file <none>
I20250625 07:58:32.012857 8947 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:32.013073 8947 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:32.021070 8947 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250625 07:58:32.025942 8972 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:32.027153 8947 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 07:58:32.027519 8947 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cc8ab07892f84f6f910e5ee36cab6300"
format_stamp: "Formatted at 2025-06-25 07:58:19 on dist-test-slave-8zlz"
I20250625 07:58:32.029616 8947 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:32.078682 8947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:32.080188 8947 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:32.080629 8947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:32.083164 8947 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:32.088786 8979 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 07:58:32.096179 8947 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 07:58:32.096433 8947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250625 07:58:32.096699 8947 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 07:58:32.101300 8947 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 07:58:32.101497 8947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250625 07:58:32.101876 8979 tablet_bootstrap.cc:492] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap starting.
I20250625 07:58:32.157944 8979 log.cc:826] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:32.259074 8979 tablet_bootstrap.cc:492] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:32.259841 8979 tablet_bootstrap.cc:492] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap complete.
I20250625 07:58:32.260938 8979 ts_tablet_manager.cc:1397] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Time spent bootstrapping tablet: real 0.159s user 0.117s sys 0.038s
I20250625 07:58:32.261152 8947 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.66:45573
I20250625 07:58:32.261322 9086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.66:45573 every 8 connection(s)
I20250625 07:58:32.263757 8947 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 07:58:32.274029 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 8947
I20250625 07:58:32.275424 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.67:45551
--local_ip_for_outbound_sockets=127.5.85.67
--tserver_master_addrs=127.5.85.126:43881
--webserver_port=36077
--webserver_interface=127.5.85.67
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:58:32.278473 8979 raft_consensus.cc:357] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:32.281230 8979 raft_consensus.cc:738] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8ab07892f84f6f910e5ee36cab6300, State: Initialized, Role: FOLLOWER
I20250625 07:58:32.282056 8979 consensus_queue.cc:260] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:32.282661 8979 raft_consensus.cc:397] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:32.282949 8979 raft_consensus.cc:491] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:32.283421 8979 raft_consensus.cc:3058] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 1 FOLLOWER]: Advancing to term 2
I20250625 07:58:32.287460 9087 heartbeater.cc:344] Connected to a master server at 127.5.85.126:43881
I20250625 07:58:32.287937 9087 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:32.289113 9087 heartbeater.cc:507] Master 127.5.85.126:43881 requested a full tablet report, sending...
I20250625 07:58:32.290584 8979 raft_consensus.cc:513] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:32.291311 8979 leader_election.cc:304] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc8ab07892f84f6f910e5ee36cab6300; no voters:
I20250625 07:58:32.293771 8979 leader_election.cc:290] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 07:58:32.294795 9092 raft_consensus.cc:2802] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 07:58:32.300045 9092 raft_consensus.cc:695] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEADER]: Becoming Leader. State: Replica: cc8ab07892f84f6f910e5ee36cab6300, State: Running, Role: LEADER
I20250625 07:58:32.300740 8756 ts_manager.cc:194] Registered new tserver with Master: cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573)
I20250625 07:58:32.300931 9092 consensus_queue.cc:237] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:32.304179 8756 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.66:35257
I20250625 07:58:32.305876 8979 ts_tablet_manager.cc:1428] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Time spent starting tablet: real 0.045s user 0.035s sys 0.009s
I20250625 07:58:32.308318 9087 heartbeater.cc:499] Master 127.5.85.126:43881 was elected leader, sending a full tablet report...
I20250625 07:58:32.311913 8756 catalog_manager.cc:5582] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "cc8ab07892f84f6f910e5ee36cab6300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } health_report { overall_health: HEALTHY } } }
W20250625 07:58:32.581194 9091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:32.581688 9091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:32.582185 9091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:32.620396 9091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:32.621706 9091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.67
I20250625 07:58:32.662217 9091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.67:45551
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.85.67
--webserver_port=36077
--tserver_master_addrs=127.5.85.126:43881
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.67
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:32.663560 9091 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:32.665215 9091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:32.680559 9106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:32.912647 8898 consensus_queue.cc:237] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } }
I20250625 07:58:32.920436 9117 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index -1 to 10, NON_VOTER cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } } }
I20250625 07:58:32.931505 8743 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 557364b8ffd8492f8f2a2a616a7c127e with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250625 07:58:32.934276 8756 catalog_manager.cc:5582] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb reported cstate change: config changed from index -1 to 10, NON_VOTER cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) added. New cstate: current_term: 2 leader_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250625 07:58:32.941560 8832 consensus_peers.cc:487] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb -> Peer cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573): Couldn't send request to peer cc8ab07892f84f6f910e5ee36cab6300. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 557364b8ffd8492f8f2a2a616a7c127e. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:58:33.458019 9124 ts_tablet_manager.cc:927] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Initiating tablet copy from peer f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65:43657)
I20250625 07:58:33.460758 9124 tablet_copy_client.cc:323] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: tablet copy: Beginning tablet copy session from remote peer at address 127.5.85.65:43657
I20250625 07:58:33.489420 8918 tablet_copy_service.cc:140] P f9b866b17f3446a6bd5e380d224c6aeb: Received BeginTabletCopySession request for tablet 557364b8ffd8492f8f2a2a616a7c127e from peer cc8ab07892f84f6f910e5ee36cab6300 ({username='slave'} at 127.5.85.66:42077)
I20250625 07:58:33.490149 8918 tablet_copy_service.cc:161] P f9b866b17f3446a6bd5e380d224c6aeb: Beginning new tablet copy session on tablet 557364b8ffd8492f8f2a2a616a7c127e from peer cc8ab07892f84f6f910e5ee36cab6300 at {username='slave'} at 127.5.85.66:42077: session id = cc8ab07892f84f6f910e5ee36cab6300-557364b8ffd8492f8f2a2a616a7c127e
I20250625 07:58:33.498039 8918 tablet_copy_source_session.cc:215] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 07:58:33.502856 9124 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557364b8ffd8492f8f2a2a616a7c127e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:33.513545 9124 tablet_copy_client.cc:806] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: tablet copy: Starting download of 0 data blocks...
I20250625 07:58:33.514140 9124 tablet_copy_client.cc:670] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: tablet copy: Starting download of 1 WAL segments...
I20250625 07:58:33.519555 9124 tablet_copy_client.cc:538] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 07:58:33.525681 9124 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap starting.
I20250625 07:58:33.639609 9124 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:33.640403 9124 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap complete.
I20250625 07:58:33.640944 9124 ts_tablet_manager.cc:1397] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Time spent bootstrapping tablet: real 0.115s user 0.099s sys 0.015s
I20250625 07:58:33.643143 9124 raft_consensus.cc:357] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } }
I20250625 07:58:33.643719 9124 raft_consensus.cc:738] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cc8ab07892f84f6f910e5ee36cab6300, State: Initialized, Role: LEARNER
I20250625 07:58:33.644243 9124 consensus_queue.cc:260] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } }
I20250625 07:58:33.647346 9124 ts_tablet_manager.cc:1428] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20250625 07:58:33.649005 8918 tablet_copy_service.cc:342] P f9b866b17f3446a6bd5e380d224c6aeb: Request end of tablet copy session cc8ab07892f84f6f910e5ee36cab6300-557364b8ffd8492f8f2a2a616a7c127e received from {username='slave'} at 127.5.85.66:42077
I20250625 07:58:33.649417 8918 tablet_copy_service.cc:434] P f9b866b17f3446a6bd5e380d224c6aeb: ending tablet copy session cc8ab07892f84f6f910e5ee36cab6300-557364b8ffd8492f8f2a2a616a7c127e on tablet 557364b8ffd8492f8f2a2a616a7c127e with peer cc8ab07892f84f6f910e5ee36cab6300
W20250625 07:58:32.682256 9107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:32.683604 9091 server_base.cc:1048] running on GCE node
W20250625 07:58:32.682324 9109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:33.792484 9091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:33.794713 9091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:33.796090 9091 hybrid_clock.cc:648] HybridClock initialized: now 1750838313796038 us; error 40 us; skew 500 ppm
I20250625 07:58:33.796856 9091 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:33.802794 9091 webserver.cc:469] Webserver started at http://127.5.85.67:36077/ using document root <none> and password file <none>
I20250625 07:58:33.803691 9091 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:33.803926 9091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:33.811910 9091 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20250625 07:58:33.816457 9134 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:33.817544 9091 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 07:58:33.817873 9091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "92b13ef756c54976a43d1e79ab68a602"
format_stamp: "Formatted at 2025-06-25 07:58:20 on dist-test-slave-8zlz"
I20250625 07:58:33.819842 9091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:33.838043 9042 raft_consensus.cc:1215] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250625 07:58:33.888216 9091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:33.889632 9091 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:33.890035 9091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:33.892486 9091 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:33.897994 9143 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 07:58:33.905602 9091 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 07:58:33.905928 9091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250625 07:58:33.906262 9091 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 07:58:33.911104 9091 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 07:58:33.911366 9091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.003s
I20250625 07:58:33.911687 9143 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Bootstrap starting.
I20250625 07:58:33.965600 9143 log.cc:826] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:34.069288 9091 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.67:45551
I20250625 07:58:34.069398 9250 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.67:45551 every 8 connection(s)
I20250625 07:58:34.070299 9143 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:34.071251 9143 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Bootstrap complete.
I20250625 07:58:34.072669 9091 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 07:58:34.072777 9143 ts_tablet_manager.cc:1397] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Time spent bootstrapping tablet: real 0.161s user 0.113s sys 0.045s
I20250625 07:58:34.081683 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 9091
I20250625 07:58:34.091845 9143 raft_consensus.cc:357] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } }
I20250625 07:58:34.094245 9143 raft_consensus.cc:738] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b13ef756c54976a43d1e79ab68a602, State: Initialized, Role: FOLLOWER
I20250625 07:58:34.095225 9143 consensus_queue.cc:260] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } }
I20250625 07:58:34.095856 9143 raft_consensus.cc:397] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:34.096177 9143 raft_consensus.cc:491] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:34.096544 9143 raft_consensus.cc:3058] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 1 FOLLOWER]: Advancing to term 2
I20250625 07:58:34.099830 9251 heartbeater.cc:344] Connected to a master server at 127.5.85.126:43881
I20250625 07:58:34.100220 9251 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:34.101112 9251 heartbeater.cc:507] Master 127.5.85.126:43881 requested a full tablet report, sending...
I20250625 07:58:34.103219 9143 raft_consensus.cc:513] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } }
I20250625 07:58:34.103899 9143 leader_election.cc:304] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92b13ef756c54976a43d1e79ab68a602; no voters:
I20250625 07:58:34.106256 9143 leader_election.cc:290] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 07:58:34.106681 9257 raft_consensus.cc:2802] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 07:58:34.109021 9257 raft_consensus.cc:695] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 2 LEADER]: Becoming Leader. State: Replica: 92b13ef756c54976a43d1e79ab68a602, State: Running, Role: LEADER
I20250625 07:58:34.109867 9257 consensus_queue.cc:237] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } }
I20250625 07:58:34.112418 9143 ts_tablet_manager.cc:1428] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Time spent starting tablet: real 0.039s user 0.034s sys 0.001s
I20250625 07:58:34.114495 8755 ts_manager.cc:194] Registered new tserver with Master: 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551)
I20250625 07:58:34.117342 8755 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.67:57563
I20250625 07:58:34.120252 9251 heartbeater.cc:499] Master 127.5.85.126:43881 was elected leader, sending a full tablet report...
I20250625 07:58:34.123050 8755 catalog_manager.cc:5582] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 reported cstate change: term changed from 0 to 2, leader changed from <none> to 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67), VOTER 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) added. New cstate: current_term: 2 leader_uuid: "92b13ef756c54976a43d1e79ab68a602" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } health_report { overall_health: HEALTHY } } }
I20250625 07:58:34.128119 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:58:34.132364 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
W20250625 07:58:34.137231 5461 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 557364b8ffd8492f8f2a2a616a7c127e: tablet_id: "557364b8ffd8492f8f2a2a616a7c127e" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: LEARNER }
I20250625 07:58:34.147341 9206 consensus_queue.cc:237] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: true } }
I20250625 07:58:34.150657 9258 raft_consensus.cc:2953] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: true } } }
I20250625 07:58:34.157557 8740 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5f98cc844fb24ece82373c7f157fa07c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250625 07:58:34.160845 9140 consensus_peers.cc:487] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 -> Peer f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65:43657): Couldn't send request to peer f9b866b17f3446a6bd5e380d224c6aeb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f98cc844fb24ece82373c7f157fa07c. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:58:34.160674 8755 catalog_manager.cc:5582] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 reported cstate change: config changed from index -1 to 8, NON_VOTER f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65) added. New cstate: current_term: 2 leader_uuid: "92b13ef756c54976a43d1e79ab68a602" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250625 07:58:34.169817 9206 consensus_queue.cc:237] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: true } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } }
I20250625 07:58:34.173007 9258 raft_consensus.cc:2953] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: true } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } } }
W20250625 07:58:34.175093 9140 consensus_peers.cc:487] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 -> Peer f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65:43657): Couldn't send request to peer f9b866b17f3446a6bd5e380d224c6aeb. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f98cc844fb24ece82373c7f157fa07c. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:58:34.180034 8740 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 5f98cc844fb24ece82373c7f157fa07c with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250625 07:58:34.183346 9140 consensus_peers.cc:487] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 -> Peer cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573): Couldn't send request to peer cc8ab07892f84f6f910e5ee36cab6300. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5f98cc844fb24ece82373c7f157fa07c. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:58:34.183211 8755 catalog_manager.cc:5582] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 reported cstate change: config changed from index 8 to 9, NON_VOTER cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) added. New cstate: current_term: 2 leader_uuid: "92b13ef756c54976a43d1e79ab68a602" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250625 07:58:34.418716 9271 raft_consensus.cc:1062] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: attempting to promote NON_VOTER cc8ab07892f84f6f910e5ee36cab6300 to VOTER
I20250625 07:58:34.420156 9271 consensus_queue.cc:237] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:34.424222 9042 raft_consensus.cc:1273] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEARNER]: Refusing update from remote peer f9b866b17f3446a6bd5e380d224c6aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250625 07:58:34.425571 9270 consensus_queue.cc:1035] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250625 07:58:34.431671 9271 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } }
I20250625 07:58:34.433254 9042 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } }
I20250625 07:58:34.443078 8755 catalog_manager.cc:5582] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb reported cstate change: config changed from index 10 to 11, cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250625 07:58:34.453347 8898 consensus_queue.cc:237] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: NON_VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: true } }
I20250625 07:58:34.458451 9042 raft_consensus.cc:1273] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Refusing update from remote peer f9b866b17f3446a6bd5e380d224c6aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250625 07:58:34.459939 9274 consensus_queue.cc:1035] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250625 07:58:34.465976 9274 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: NON_VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: true } } }
I20250625 07:58:34.466987 9042 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, NON_VOTER 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) added. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: NON_VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: true } } }
W20250625 07:58:34.469739 8829 consensus_peers.cc:487] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb -> Peer 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551): Couldn't send request to peer 92b13ef756c54976a43d1e79ab68a602. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 557364b8ffd8492f8f2a2a616a7c127e. This is attempt 1: this message will repeat every 5th retry.
I20250625 07:58:34.474321 8743 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 557364b8ffd8492f8f2a2a616a7c127e with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250625 07:58:34.476819 8755 catalog_manager.cc:5582] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 reported cstate change: config changed from index 11 to 12, NON_VOTER 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) added. New cstate: current_term: 2 leader_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: NON_VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: true } } }
I20250625 07:58:34.578583 9277 ts_tablet_manager.cc:927] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Initiating tablet copy from peer 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551)
I20250625 07:58:34.579980 9277 tablet_copy_client.cc:323] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: tablet copy: Beginning tablet copy session from remote peer at address 127.5.85.67:45551
I20250625 07:58:34.589936 9226 tablet_copy_service.cc:140] P 92b13ef756c54976a43d1e79ab68a602: Received BeginTabletCopySession request for tablet 5f98cc844fb24ece82373c7f157fa07c from peer cc8ab07892f84f6f910e5ee36cab6300 ({username='slave'} at 127.5.85.66:45837)
I20250625 07:58:34.590422 9226 tablet_copy_service.cc:161] P 92b13ef756c54976a43d1e79ab68a602: Beginning new tablet copy session on tablet 5f98cc844fb24ece82373c7f157fa07c from peer cc8ab07892f84f6f910e5ee36cab6300 at {username='slave'} at 127.5.85.66:45837: session id = cc8ab07892f84f6f910e5ee36cab6300-5f98cc844fb24ece82373c7f157fa07c
I20250625 07:58:34.594813 9280 ts_tablet_manager.cc:927] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Initiating tablet copy from peer 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551)
I20250625 07:58:34.595479 9226 tablet_copy_source_session.cc:215] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 07:58:34.597620 9280 tablet_copy_client.cc:323] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: tablet copy: Beginning tablet copy session from remote peer at address 127.5.85.67:45551
I20250625 07:58:34.599344 9277 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f98cc844fb24ece82373c7f157fa07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:34.599524 9226 tablet_copy_service.cc:140] P 92b13ef756c54976a43d1e79ab68a602: Received BeginTabletCopySession request for tablet 5f98cc844fb24ece82373c7f157fa07c from peer f9b866b17f3446a6bd5e380d224c6aeb ({username='slave'} at 127.5.85.65:34081)
I20250625 07:58:34.600014 9226 tablet_copy_service.cc:161] P 92b13ef756c54976a43d1e79ab68a602: Beginning new tablet copy session on tablet 5f98cc844fb24ece82373c7f157fa07c from peer f9b866b17f3446a6bd5e380d224c6aeb at {username='slave'} at 127.5.85.65:34081: session id = f9b866b17f3446a6bd5e380d224c6aeb-5f98cc844fb24ece82373c7f157fa07c
I20250625 07:58:34.604318 9226 tablet_copy_source_session.cc:215] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 07:58:34.607120 9280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5f98cc844fb24ece82373c7f157fa07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:34.611120 9277 tablet_copy_client.cc:806] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: tablet copy: Starting download of 0 data blocks...
I20250625 07:58:34.611626 9277 tablet_copy_client.cc:670] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: tablet copy: Starting download of 1 WAL segments...
I20250625 07:58:34.616127 9277 tablet_copy_client.cc:538] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 07:58:34.620854 9280 tablet_copy_client.cc:806] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: tablet copy: Starting download of 0 data blocks...
I20250625 07:58:34.621405 9280 tablet_copy_client.cc:670] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: tablet copy: Starting download of 1 WAL segments...
I20250625 07:58:34.623843 9277 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap starting.
I20250625 07:58:34.625939 9280 tablet_copy_client.cc:538] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 07:58:34.634483 9280 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap starting.
I20250625 07:58:34.729110 9277 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:34.729703 9277 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap complete.
I20250625 07:58:34.730126 9277 ts_tablet_manager.cc:1397] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Time spent bootstrapping tablet: real 0.106s user 0.094s sys 0.010s
I20250625 07:58:34.731585 9277 raft_consensus.cc:357] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: true } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } }
I20250625 07:58:34.732035 9277 raft_consensus.cc:738] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: cc8ab07892f84f6f910e5ee36cab6300, State: Initialized, Role: LEARNER
I20250625 07:58:34.732429 9277 consensus_queue.cc:260] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: true } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } }
I20250625 07:58:34.733755 9277 ts_tablet_manager.cc:1428] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250625 07:58:34.735678 9226 tablet_copy_service.cc:342] P 92b13ef756c54976a43d1e79ab68a602: Request end of tablet copy session cc8ab07892f84f6f910e5ee36cab6300-5f98cc844fb24ece82373c7f157fa07c received from {username='slave'} at 127.5.85.66:45837
I20250625 07:58:34.736102 9226 tablet_copy_service.cc:434] P 92b13ef756c54976a43d1e79ab68a602: ending tablet copy session cc8ab07892f84f6f910e5ee36cab6300-5f98cc844fb24ece82373c7f157fa07c on tablet 5f98cc844fb24ece82373c7f157fa07c with peer cc8ab07892f84f6f910e5ee36cab6300
I20250625 07:58:34.737488 9280 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:34.738086 9280 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap complete.
I20250625 07:58:34.738616 9280 ts_tablet_manager.cc:1397] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Time spent bootstrapping tablet: real 0.104s user 0.088s sys 0.016s
I20250625 07:58:34.740672 9280 raft_consensus.cc:357] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: true } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } }
I20250625 07:58:34.741128 9280 raft_consensus.cc:738] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f9b866b17f3446a6bd5e380d224c6aeb, State: Initialized, Role: LEARNER
I20250625 07:58:34.741540 9280 consensus_queue.cc:260] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: NON_VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: true } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } }
I20250625 07:58:34.742918 9280 ts_tablet_manager.cc:1428] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250625 07:58:34.744298 9226 tablet_copy_service.cc:342] P 92b13ef756c54976a43d1e79ab68a602: Request end of tablet copy session f9b866b17f3446a6bd5e380d224c6aeb-5f98cc844fb24ece82373c7f157fa07c received from {username='slave'} at 127.5.85.65:34081
I20250625 07:58:34.744581 9226 tablet_copy_service.cc:434] P 92b13ef756c54976a43d1e79ab68a602: ending tablet copy session f9b866b17f3446a6bd5e380d224c6aeb-5f98cc844fb24ece82373c7f157fa07c on tablet 5f98cc844fb24ece82373c7f157fa07c with peer f9b866b17f3446a6bd5e380d224c6aeb
I20250625 07:58:34.959175 9284 ts_tablet_manager.cc:927] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Initiating tablet copy from peer f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65:43657)
I20250625 07:58:34.960975 9284 tablet_copy_client.cc:323] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: tablet copy: Beginning tablet copy session from remote peer at address 127.5.85.65:43657
I20250625 07:58:34.962554 8918 tablet_copy_service.cc:140] P f9b866b17f3446a6bd5e380d224c6aeb: Received BeginTabletCopySession request for tablet 557364b8ffd8492f8f2a2a616a7c127e from peer 92b13ef756c54976a43d1e79ab68a602 ({username='slave'} at 127.5.85.67:51245)
I20250625 07:58:34.962944 8918 tablet_copy_service.cc:161] P f9b866b17f3446a6bd5e380d224c6aeb: Beginning new tablet copy session on tablet 557364b8ffd8492f8f2a2a616a7c127e from peer 92b13ef756c54976a43d1e79ab68a602 at {username='slave'} at 127.5.85.67:51245: session id = 92b13ef756c54976a43d1e79ab68a602-557364b8ffd8492f8f2a2a616a7c127e
I20250625 07:58:34.967214 8918 tablet_copy_source_session.cc:215] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Tablet Copy: opened 0 blocks and 1 log segments
I20250625 07:58:34.969775 9284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 557364b8ffd8492f8f2a2a616a7c127e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:58:34.982597 9284 tablet_copy_client.cc:806] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: tablet copy: Starting download of 0 data blocks...
I20250625 07:58:34.983163 9284 tablet_copy_client.cc:670] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: tablet copy: Starting download of 1 WAL segments...
I20250625 07:58:34.986714 9284 tablet_copy_client.cc:538] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250625 07:58:34.992210 9284 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Bootstrap starting.
I20250625 07:58:35.066610 8898 raft_consensus.cc:1215] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250625 07:58:35.087005 9284 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:35.087564 9284 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Bootstrap complete.
I20250625 07:58:35.087986 9284 ts_tablet_manager.cc:1397] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Time spent bootstrapping tablet: real 0.096s user 0.090s sys 0.005s
I20250625 07:58:35.089674 9284 raft_consensus.cc:357] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: NON_VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: true } }
I20250625 07:58:35.090106 9284 raft_consensus.cc:738] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 92b13ef756c54976a43d1e79ab68a602, State: Initialized, Role: LEARNER
I20250625 07:58:35.090516 9284 consensus_queue.cc:260] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: NON_VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: true } }
I20250625 07:58:35.092051 9284 ts_tablet_manager.cc:1428] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250625 07:58:35.093832 8918 tablet_copy_service.cc:342] P f9b866b17f3446a6bd5e380d224c6aeb: Request end of tablet copy session 92b13ef756c54976a43d1e79ab68a602-557364b8ffd8492f8f2a2a616a7c127e received from {username='slave'} at 127.5.85.67:51245
I20250625 07:58:35.094228 8918 tablet_copy_service.cc:434] P f9b866b17f3446a6bd5e380d224c6aeb: ending tablet copy session 92b13ef756c54976a43d1e79ab68a602-557364b8ffd8492f8f2a2a616a7c127e on tablet 557364b8ffd8492f8f2a2a616a7c127e with peer 92b13ef756c54976a43d1e79ab68a602
I20250625 07:58:35.141647 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f9b866b17f3446a6bd5e380d224c6aeb to finish bootstrapping
I20250625 07:58:35.154784 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cc8ab07892f84f6f910e5ee36cab6300 to finish bootstrapping
I20250625 07:58:35.164314 8742 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 557364b8ffd8492f8f2a2a616a7c127e with cas_config_opid_index 10: aborting the task: latest config opid_index 12; task opid_index 10
I20250625 07:58:35.168169 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 92b13ef756c54976a43d1e79ab68a602 to finish bootstrapping
I20250625 07:58:35.202142 9042 raft_consensus.cc:1215] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250625 07:58:35.423955 9206 raft_consensus.cc:1215] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.11->[2.12-2.12] Dedup: 2.12->[]
I20250625 07:58:35.436323 9186 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 07:58:35.437498 8878 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 07:58:35.437696 9022 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 07:58:35.582917 9288 raft_consensus.cc:1062] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: attempting to promote NON_VOTER f9b866b17f3446a6bd5e380d224c6aeb to VOTER
I20250625 07:58:35.584936 9288 consensus_queue.cc:237] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } }
I20250625 07:58:35.599403 9042 raft_consensus.cc:1273] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEARNER]: Refusing update from remote peer 92b13ef756c54976a43d1e79ab68a602: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250625 07:58:35.601047 9258 consensus_queue.cc:1035] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250625 07:58:35.610361 8898 raft_consensus.cc:1273] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 LEARNER]: Refusing update from remote peer 92b13ef756c54976a43d1e79ab68a602: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250625 07:58:35.612026 9258 consensus_queue.cc:1035] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250625 07:58:35.632108 9258 raft_consensus.cc:2953] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } } }
I20250625 07:58:35.640903 8897 raft_consensus.cc:2953] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } } }
I20250625 07:58:35.647415 8756 catalog_manager.cc:5582] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 reported cstate change: config changed from index 9 to 10, f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "92b13ef756c54976a43d1e79ab68a602" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250625 07:58:35.662997 9287 raft_consensus.cc:1062] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: attempting to promote NON_VOTER cc8ab07892f84f6f910e5ee36cab6300 to VOTER
I20250625 07:58:35.664674 9287 consensus_queue.cc:237] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:35.667990 9041 raft_consensus.cc:2953] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: NON_VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: true } } }
I20250625 07:58:35.677376 8897 raft_consensus.cc:1273] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Refusing update from remote peer 92b13ef756c54976a43d1e79ab68a602: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250625 07:58:35.679692 9325 consensus_queue.cc:1035] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250625 07:58:35.686452 9258 raft_consensus.cc:2953] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } }
I20250625 07:58:35.688144 8898 raft_consensus.cc:2953] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } }
I20250625 07:58:35.695554 9040 raft_consensus.cc:1273] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 LEARNER]: Refusing update from remote peer 92b13ef756c54976a43d1e79ab68a602: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250625 07:58:35.703795 8756 catalog_manager.cc:5582] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 reported cstate change: config changed from index 10 to 11, cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "92b13ef756c54976a43d1e79ab68a602" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250625 07:58:35.705500 9288 consensus_queue.cc:1035] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250625 07:58:35.712147 9040 raft_consensus.cc:2953] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } }
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
faf56aa493c943eda6ba3b0c6587d216 | 127.5.85.126:43881 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.5.85.84:37239 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
92b13ef756c54976a43d1e79ab68a602 | 127.5.85.67:45551 | HEALTHY | <none> | 1 | 0
cc8ab07892f84f6f910e5ee36cab6300 | 127.5.85.66:45573 | HEALTHY | <none> | 1 | 0
f9b866b17f3446a6bd5e380d224c6aeb | 127.5.85.65:43657 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.85.65 | experimental | 127.5.85.65:43657
local_ip_for_outbound_sockets | 127.5.85.66 | experimental | 127.5.85.66:45573
local_ip_for_outbound_sockets | 127.5.85.67 | experimental | 127.5.85.67:45551
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.5.85.65:43657
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.5.85.66:45573
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.5.85.67:45551
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.5.85.84:37239 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250625 07:58:35.760710 5461 log_verifier.cc:126] Checking tablet 0534efd80026410aae53e615ef097408
I20250625 07:58:35.792833 5461 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0534efd80026410aae53e615ef097408
I20250625 07:58:35.793076 5461 log_verifier.cc:126] Checking tablet 557364b8ffd8492f8f2a2a616a7c127e
I20250625 07:58:35.896627 5461 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 557364b8ffd8492f8f2a2a616a7c127e
I20250625 07:58:35.896867 5461 log_verifier.cc:126] Checking tablet 5f98cc844fb24ece82373c7f157fa07c
I20250625 07:58:35.943619 9319 raft_consensus.cc:1062] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: attempting to promote NON_VOTER 92b13ef756c54976a43d1e79ab68a602 to VOTER
I20250625 07:58:35.944995 9319 consensus_queue.cc:237] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:35.949872 9206 raft_consensus.cc:1273] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 2 LEARNER]: Refusing update from remote peer f9b866b17f3446a6bd5e380d224c6aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250625 07:58:35.951169 9319 consensus_queue.cc:1035] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250625 07:58:35.951387 9040 raft_consensus.cc:1273] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Refusing update from remote peer f9b866b17f3446a6bd5e380d224c6aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20250625 07:58:35.952589 9329 consensus_queue.cc:1035] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20250625 07:58:35.960587 9040 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } } }
I20250625 07:58:35.960918 9206 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 2 FOLLOWER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } } }
I20250625 07:58:35.958912 9319 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 LEADER]: Committing config change with OpId 2.13: config changed from index 12 to 13, 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } } }
I20250625 07:58:35.971349 8754 catalog_manager.cc:5582] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 reported cstate change: config changed from index 12 to 13, 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } } }
I20250625 07:58:35.991164 5461 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 5f98cc844fb24ece82373c7f157fa07c
I20250625 07:58:35.991575 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 8723
I20250625 07:58:36.015786 5461 minidump.cc:252] Setting minidump size limit to 20M
I20250625 07:58:36.017001 5461 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:36.017972 5461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:36.028267 9336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:36.028723 9335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:36.029167 9338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:36.029522 5461 server_base.cc:1048] running on GCE node
I20250625 07:58:36.123890 5461 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250625 07:58:36.124106 5461 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250625 07:58:36.124282 5461 hybrid_clock.cc:648] HybridClock initialized: now 1750838316124263 us; error 0 us; skew 500 ppm
I20250625 07:58:36.124940 5461 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:36.128046 5461 webserver.cc:469] Webserver started at http://0.0.0.0:36671/ using document root <none> and password file <none>
I20250625 07:58:36.128801 5461 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:36.128988 5461 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:36.133963 5461 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20250625 07:58:36.137509 9343 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:36.138585 5461 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250625 07:58:36.138964 5461 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "faf56aa493c943eda6ba3b0c6587d216"
format_stamp: "Formatted at 2025-06-25 07:58:15 on dist-test-slave-8zlz"
I20250625 07:58:36.140638 5461 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:36.152990 5461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:36.154451 5461 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:36.154925 5461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:36.163465 5461 sys_catalog.cc:263] Verifying existing consensus state
W20250625 07:58:36.167124 5461 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.5.85.126:43881 exists but no master address supplied!
I20250625 07:58:36.169224 5461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Bootstrap starting.
I20250625 07:58:36.210745 5461 log.cc:826] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Log is configured to *not* fsync() on all Append() calls
W20250625 07:58:36.234308 9251 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:43881 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:43881: connect: Connection refused (error 111)
I20250625 07:58:36.274748 5461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:36.275493 5461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Bootstrap complete.
I20250625 07:58:36.289027 5461 raft_consensus.cc:357] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:36.289619 5461 raft_consensus.cc:738] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf56aa493c943eda6ba3b0c6587d216, State: Initialized, Role: FOLLOWER
I20250625 07:58:36.290342 5461 consensus_queue.cc:260] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:36.290858 5461 raft_consensus.cc:397] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:36.291105 5461 raft_consensus.cc:491] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:36.291432 5461 raft_consensus.cc:3058] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 3 FOLLOWER]: Advancing to term 4
I20250625 07:58:36.297076 5461 raft_consensus.cc:513] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:36.297827 5461 leader_election.cc:304] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faf56aa493c943eda6ba3b0c6587d216; no voters:
I20250625 07:58:36.298995 5461 leader_election.cc:290] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250625 07:58:36.299253 9353 raft_consensus.cc:2802] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 4 FOLLOWER]: Leader election won for term 4
I20250625 07:58:36.300498 9353 raft_consensus.cc:695] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 4 LEADER]: Becoming Leader. State: Replica: faf56aa493c943eda6ba3b0c6587d216, State: Running, Role: LEADER
I20250625 07:58:36.301241 9353 consensus_queue.cc:237] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:36.308854 9354 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "faf56aa493c943eda6ba3b0c6587d216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } } }
I20250625 07:58:36.309067 9355 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faf56aa493c943eda6ba3b0c6587d216. Latest consensus state: current_term: 4 leader_uuid: "faf56aa493c943eda6ba3b0c6587d216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } } }
I20250625 07:58:36.309558 9355 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: This master's current role is: LEADER
I20250625 07:58:36.309558 9354 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: This master's current role is: LEADER
I20250625 07:58:36.342700 5461 tablet_replica.cc:331] stopping tablet replica
I20250625 07:58:36.343331 5461 raft_consensus.cc:2241] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 4 LEADER]: Raft consensus shutting down.
I20250625 07:58:36.343778 5461 raft_consensus.cc:2270] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250625 07:58:36.346103 5461 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250625 07:58:36.346618 5461 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250625 07:58:36.372828 5461 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250625 07:58:37.004648 8943 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:43881 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:43881: connect: Connection refused (error 111)
W20250625 07:58:37.010088 9087 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:43881 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:43881: connect: Connection refused (error 111)
I20250625 07:58:41.417943 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 8793
I20250625 07:58:41.440555 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 8947
I20250625 07:58:41.465750 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 9091
I20250625 07:58:41.490041 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:43881
--webserver_interface=127.5.85.126
--webserver_port=44849
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:43881 with env {}
W20250625 07:58:41.799213 9433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:41.799770 9433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:41.800175 9433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:41.831260 9433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:58:41.831566 9433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:41.831769 9433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:58:41.831965 9433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:58:41.868625 9433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:43881
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:43881
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=44849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:41.870496 9433 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:41.872390 9433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:41.887457 9442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:41.887437 9440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:41.887451 9439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:42.976670 9441 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:58:42.976789 9433 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:58:42.980464 9433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:42.983309 9433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:42.984699 9433 hybrid_clock.cc:648] HybridClock initialized: now 1750838322984665 us; error 53 us; skew 500 ppm
I20250625 07:58:42.985423 9433 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:42.991602 9433 webserver.cc:469] Webserver started at http://127.5.85.126:44849/ using document root <none> and password file <none>
I20250625 07:58:42.992455 9433 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:42.992646 9433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:42.999925 9433 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250625 07:58:43.004113 9450 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:43.005067 9433 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250625 07:58:43.005355 9433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "faf56aa493c943eda6ba3b0c6587d216"
format_stamp: "Formatted at 2025-06-25 07:58:15 on dist-test-slave-8zlz"
I20250625 07:58:43.007455 9433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:43.064782 9433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:43.066449 9433 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:43.066974 9433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:43.158263 9433 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:43881
I20250625 07:58:43.158344 9501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:43881 every 8 connection(s)
I20250625 07:58:43.162367 9433 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 07:58:43.172308 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 9433
I20250625 07:58:43.173059 9502 sys_catalog.cc:263] Verifying existing consensus state
I20250625 07:58:43.173950 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:43657
--local_ip_for_outbound_sockets=127.5.85.65
--tserver_master_addrs=127.5.85.126:43881
--webserver_port=35295
--webserver_interface=127.5.85.65
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:58:43.177939 9502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Bootstrap starting.
I20250625 07:58:43.188354 9502 log.cc:826] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:43.289575 9502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:43.290297 9502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Bootstrap complete.
I20250625 07:58:43.308490 9502 raft_consensus.cc:357] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:43.310370 9502 raft_consensus.cc:738] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: faf56aa493c943eda6ba3b0c6587d216, State: Initialized, Role: FOLLOWER
I20250625 07:58:43.311151 9502 consensus_queue.cc:260] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:43.311600 9502 raft_consensus.cc:397] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:43.311834 9502 raft_consensus.cc:491] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:43.312090 9502 raft_consensus.cc:3058] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 5 FOLLOWER]: Advancing to term 6
I20250625 07:58:43.316985 9502 raft_consensus.cc:513] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:43.317554 9502 leader_election.cc:304] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: faf56aa493c943eda6ba3b0c6587d216; no voters:
I20250625 07:58:43.319216 9502 leader_election.cc:290] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250625 07:58:43.319706 9506 raft_consensus.cc:2802] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 6 FOLLOWER]: Leader election won for term 6
I20250625 07:58:43.322489 9506 raft_consensus.cc:695] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [term 6 LEADER]: Becoming Leader. State: Replica: faf56aa493c943eda6ba3b0c6587d216, State: Running, Role: LEADER
I20250625 07:58:43.323558 9502 sys_catalog.cc:564] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:58:43.323344 9506 consensus_queue.cc:237] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } }
I20250625 07:58:43.332778 9507 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "faf56aa493c943eda6ba3b0c6587d216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } } }
I20250625 07:58:43.333488 9507 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: This master's current role is: LEADER
I20250625 07:58:43.335314 9508 sys_catalog.cc:455] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: SysCatalogTable state changed. Reason: New leader faf56aa493c943eda6ba3b0c6587d216. Latest consensus state: current_term: 6 leader_uuid: "faf56aa493c943eda6ba3b0c6587d216" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "faf56aa493c943eda6ba3b0c6587d216" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 43881 } } }
I20250625 07:58:43.335937 9508 sys_catalog.cc:458] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216 [sys.catalog]: This master's current role is: LEADER
I20250625 07:58:43.340742 9513 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:58:43.352751 9513 catalog_manager.cc:671] Loaded metadata for table TestTable [id=4c0cf8a8a0074dd2a4587163f74361ce]
I20250625 07:58:43.354421 9513 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=5984b52a3e1a4ecb8e0138ebd0ffb2b6]
I20250625 07:58:43.355958 9513 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=94994e4d87fd4e79ba35dd32c106ced8]
I20250625 07:58:43.364120 9513 tablet_loader.cc:96] loaded metadata for tablet 0534efd80026410aae53e615ef097408 (table TestTable2 [id=5984b52a3e1a4ecb8e0138ebd0ffb2b6])
I20250625 07:58:43.365381 9513 tablet_loader.cc:96] loaded metadata for tablet 557364b8ffd8492f8f2a2a616a7c127e (table TestTable [id=4c0cf8a8a0074dd2a4587163f74361ce])
I20250625 07:58:43.366937 9513 tablet_loader.cc:96] loaded metadata for tablet 5f98cc844fb24ece82373c7f157fa07c (table TestTable1 [id=94994e4d87fd4e79ba35dd32c106ced8])
I20250625 07:58:43.368439 9513 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:58:43.374042 9513 catalog_manager.cc:1261] Loaded cluster ID: 93851fe16d7f4765aefd25619c599ef8
I20250625 07:58:43.374338 9513 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:58:43.382725 9513 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:58:43.385680 9523 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250625 07:58:43.388475 9513 catalog_manager.cc:5966] T 00000000000000000000000000000000 P faf56aa493c943eda6ba3b0c6587d216: Loaded TSK: 0
I20250625 07:58:43.389935 9513 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250625 07:58:43.520738 9504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:43.521242 9504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:43.521721 9504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:43.555053 9504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:43.556414 9504 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:58:43.593701 9504 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:43657
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=35295
--tserver_master_addrs=127.5.85.126:43881
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:43.595075 9504 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:43.596812 9504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:43.615710 9530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:43.617615 9504 server_base.cc:1048] running on GCE node
W20250625 07:58:43.616690 9532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:43.615769 9529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:44.704198 9504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:44.706830 9504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:44.708279 9504 hybrid_clock.cc:648] HybridClock initialized: now 1750838324708219 us; error 75 us; skew 500 ppm
I20250625 07:58:44.709115 9504 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:44.722586 9504 webserver.cc:469] Webserver started at http://127.5.85.65:35295/ using document root <none> and password file <none>
I20250625 07:58:44.723578 9504 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:44.723798 9504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:44.731972 9504 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20250625 07:58:44.736920 9539 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:44.738001 9504 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 07:58:44.738312 9504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f9b866b17f3446a6bd5e380d224c6aeb"
format_stamp: "Formatted at 2025-06-25 07:58:17 on dist-test-slave-8zlz"
I20250625 07:58:44.740278 9504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:44.793283 9504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:44.794831 9504 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:44.795249 9504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:44.798030 9504 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:44.804103 9546 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250625 07:58:44.815901 9504 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250625 07:58:44.816129 9504 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.000s sys 0.002s
I20250625 07:58:44.816411 9504 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250625 07:58:44.821740 9546 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap starting.
I20250625 07:58:44.826454 9504 ts_tablet_manager.cc:610] Registered 2 tablets
I20250625 07:58:44.826771 9504 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.006s sys 0.005s
I20250625 07:58:44.897078 9546 log.cc:826] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:45.001075 9504 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:43657
I20250625 07:58:45.001247 9653 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:43657 every 8 connection(s)
I20250625 07:58:45.004859 9504 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 07:58:45.008100 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 9504
I20250625 07:58:45.009925 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.66:45573
--local_ip_for_outbound_sockets=127.5.85.66
--tserver_master_addrs=127.5.85.126:43881
--webserver_port=34965
--webserver_interface=127.5.85.66
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:58:45.032207 9546 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:45.033313 9546 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap complete.
I20250625 07:58:45.035145 9546 ts_tablet_manager.cc:1397] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Time spent bootstrapping tablet: real 0.214s user 0.171s sys 0.039s
I20250625 07:58:45.060073 9546 raft_consensus.cc:357] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:45.063109 9546 raft_consensus.cc:738] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b866b17f3446a6bd5e380d224c6aeb, State: Initialized, Role: FOLLOWER
I20250625 07:58:45.063973 9546 consensus_queue.cc:260] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:45.076509 9546 ts_tablet_manager.cc:1428] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb: Time spent starting tablet: real 0.041s user 0.027s sys 0.013s
I20250625 07:58:45.077360 9546 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap starting.
I20250625 07:58:45.078606 9654 heartbeater.cc:344] Connected to a master server at 127.5.85.126:43881
I20250625 07:58:45.079097 9654 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:45.080286 9654 heartbeater.cc:507] Master 127.5.85.126:43881 requested a full tablet report, sending...
I20250625 07:58:45.085220 9467 ts_manager.cc:194] Registered new tserver with Master: f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65:43657)
I20250625 07:58:45.089785 9467 catalog_manager.cc:5582] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb reported cstate change: config changed from index -1 to 13, term changed from 0 to 2, VOTER 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) added, VOTER cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) added, VOTER f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65) added. New cstate: current_term: 2 committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } } }
I20250625 07:58:45.161146 9467 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:54047
I20250625 07:58:45.165629 9654 heartbeater.cc:499] Master 127.5.85.126:43881 was elected leader, sending a full tablet report...
I20250625 07:58:45.214555 9546 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:45.215452 9546 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Bootstrap complete.
I20250625 07:58:45.216849 9546 ts_tablet_manager.cc:1397] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Time spent bootstrapping tablet: real 0.140s user 0.131s sys 0.005s
I20250625 07:58:45.218955 9546 raft_consensus.cc:357] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:45.219589 9546 raft_consensus.cc:738] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9b866b17f3446a6bd5e380d224c6aeb, State: Initialized, Role: FOLLOWER
I20250625 07:58:45.220228 9546 consensus_queue.cc:260] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:45.222018 9546 ts_tablet_manager.cc:1428] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20250625 07:58:45.410820 9657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:45.411478 9657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:45.412206 9657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:45.443328 9657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:45.444213 9657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.66
I20250625 07:58:45.480682 9657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.66:45573
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.85.66
--webserver_port=34965
--tserver_master_addrs=127.5.85.126:43881
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.66
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:45.482019 9657 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:45.483705 9657 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:45.499441 9671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:46.205714 9676 raft_consensus.cc:491] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:58:46.206231 9676 raft_consensus.cc:513] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:46.208479 9676 leader_election.cc:290] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573), 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551)
W20250625 07:58:46.212172 9540 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.85.67:45551: connect: Connection refused (error 111)
W20250625 07:58:46.216233 9543 leader_election.cc:336] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573): Network error: Client connection negotiation failed: client connection to 127.5.85.66:45573: connect: Connection refused (error 111)
W20250625 07:58:46.216820 9540 leader_election.cc:336] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551): Network error: Client connection negotiation failed: client connection to 127.5.85.67:45551: connect: Connection refused (error 111)
I20250625 07:58:46.217133 9540 leader_election.cc:304] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9b866b17f3446a6bd5e380d224c6aeb; no voters: 92b13ef756c54976a43d1e79ab68a602, cc8ab07892f84f6f910e5ee36cab6300
I20250625 07:58:46.217751 9676 raft_consensus.cc:2747] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250625 07:58:46.373463 9676 raft_consensus.cc:491] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:58:46.373858 9676 raft_consensus.cc:513] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:46.375496 9676 leader_election.cc:290] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551), cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573)
W20250625 07:58:46.378985 9540 leader_election.cc:336] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551): Network error: Client connection negotiation failed: client connection to 127.5.85.67:45551: connect: Connection refused (error 111)
W20250625 07:58:46.379539 9543 leader_election.cc:336] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573): Network error: Client connection negotiation failed: client connection to 127.5.85.66:45573: connect: Connection refused (error 111)
I20250625 07:58:46.379987 9543 leader_election.cc:304] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f9b866b17f3446a6bd5e380d224c6aeb; no voters: 92b13ef756c54976a43d1e79ab68a602, cc8ab07892f84f6f910e5ee36cab6300
I20250625 07:58:46.380744 9676 raft_consensus.cc:2747] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250625 07:58:45.500463 9670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:45.499468 9673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:46.582695 9672 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:58:46.582738 9657 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:58:46.586839 9657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:46.589129 9657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:46.590479 9657 hybrid_clock.cc:648] HybridClock initialized: now 1750838326590438 us; error 49 us; skew 500 ppm
I20250625 07:58:46.591280 9657 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:46.597615 9657 webserver.cc:469] Webserver started at http://127.5.85.66:34965/ using document root <none> and password file <none>
I20250625 07:58:46.598497 9657 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:46.598732 9657 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:46.606339 9657 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20250625 07:58:46.611042 9684 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:46.612059 9657 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250625 07:58:46.612365 9657 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cc8ab07892f84f6f910e5ee36cab6300"
format_stamp: "Formatted at 2025-06-25 07:58:19 on dist-test-slave-8zlz"
I20250625 07:58:46.614306 9657 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:46.664800 9657 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:46.666339 9657 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:46.666796 9657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:46.669718 9657 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:46.675657 9691 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250625 07:58:46.690619 9657 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250625 07:58:46.690900 9657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.017s user 0.000s sys 0.002s
I20250625 07:58:46.691145 9657 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250625 07:58:46.696107 9691 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap starting.
I20250625 07:58:46.701891 9657 ts_tablet_manager.cc:610] Registered 3 tablets
I20250625 07:58:46.702096 9657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.008s sys 0.000s
I20250625 07:58:46.746280 9691 log.cc:826] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:46.849689 9691 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:46.850426 9691 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap complete.
I20250625 07:58:46.851794 9691 ts_tablet_manager.cc:1397] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Time spent bootstrapping tablet: real 0.156s user 0.129s sys 0.024s
I20250625 07:58:46.862102 9657 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.66:45573
I20250625 07:58:46.862226 9798 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.66:45573 every 8 connection(s)
I20250625 07:58:46.864603 9657 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 07:58:46.866483 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 9657
I20250625 07:58:46.868312 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.67:45551
--local_ip_for_outbound_sockets=127.5.85.67
--tserver_master_addrs=127.5.85.126:43881
--webserver_port=36077
--webserver_interface=127.5.85.67
--builtin_ntp_servers=127.5.85.84:37239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250625 07:58:46.867602 9691 raft_consensus.cc:357] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:46.870674 9691 raft_consensus.cc:738] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8ab07892f84f6f910e5ee36cab6300, State: Initialized, Role: FOLLOWER
I20250625 07:58:46.871723 9691 consensus_queue.cc:260] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:46.879878 9691 ts_tablet_manager.cc:1428] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Time spent starting tablet: real 0.028s user 0.022s sys 0.005s
I20250625 07:58:46.880618 9691 tablet_bootstrap.cc:492] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap starting.
I20250625 07:58:46.890303 9799 heartbeater.cc:344] Connected to a master server at 127.5.85.126:43881
I20250625 07:58:46.890722 9799 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:46.891625 9799 heartbeater.cc:507] Master 127.5.85.126:43881 requested a full tablet report, sending...
I20250625 07:58:46.894966 9466 ts_manager.cc:194] Registered new tserver with Master: cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573)
I20250625 07:58:46.898347 9466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.66:59723
I20250625 07:58:46.902025 9799 heartbeater.cc:499] Master 127.5.85.126:43881 was elected leader, sending a full tablet report...
I20250625 07:58:46.988982 9691 tablet_bootstrap.cc:492] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:46.989601 9691 tablet_bootstrap.cc:492] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap complete.
I20250625 07:58:46.990574 9691 ts_tablet_manager.cc:1397] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Time spent bootstrapping tablet: real 0.110s user 0.090s sys 0.016s
I20250625 07:58:46.992081 9691 raft_consensus.cc:357] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:46.992431 9691 raft_consensus.cc:738] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8ab07892f84f6f910e5ee36cab6300, State: Initialized, Role: FOLLOWER
I20250625 07:58:46.992882 9691 consensus_queue.cc:260] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:46.993337 9691 raft_consensus.cc:397] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:58:46.993649 9691 raft_consensus.cc:491] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:58:46.993940 9691 raft_consensus.cc:3058] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Advancing to term 3
I20250625 07:58:46.998945 9691 raft_consensus.cc:513] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:46.999576 9691 leader_election.cc:304] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc8ab07892f84f6f910e5ee36cab6300; no voters:
I20250625 07:58:47.000063 9691 leader_election.cc:290] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250625 07:58:47.000293 9802 raft_consensus.cc:2802] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 3 FOLLOWER]: Leader election won for term 3
I20250625 07:58:47.008641 9802 raft_consensus.cc:695] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [term 3 LEADER]: Becoming Leader. State: Replica: cc8ab07892f84f6f910e5ee36cab6300, State: Running, Role: LEADER
I20250625 07:58:47.009624 9802 consensus_queue.cc:237] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } }
I20250625 07:58:47.020248 9691 ts_tablet_manager.cc:1428] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300: Time spent starting tablet: real 0.029s user 0.012s sys 0.016s
I20250625 07:58:47.021057 9691 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap starting.
I20250625 07:58:47.032794 9466 catalog_manager.cc:5582] T 0534efd80026410aae53e615ef097408 P cc8ab07892f84f6f910e5ee36cab6300 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "cc8ab07892f84f6f910e5ee36cab6300" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } health_report { overall_health: HEALTHY } } }
I20250625 07:58:47.115231 9691 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:47.115900 9691 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Bootstrap complete.
I20250625 07:58:47.116894 9691 ts_tablet_manager.cc:1397] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Time spent bootstrapping tablet: real 0.096s user 0.082s sys 0.012s
I20250625 07:58:47.118361 9691 raft_consensus.cc:357] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:47.118868 9691 raft_consensus.cc:738] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc8ab07892f84f6f910e5ee36cab6300, State: Initialized, Role: FOLLOWER
I20250625 07:58:47.119323 9691 consensus_queue.cc:260] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:47.120615 9691 ts_tablet_manager.cc:1428] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250625 07:58:47.218739 9804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:58:47.219205 9804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:58:47.219650 9804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:58:47.252130 9804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:58:47.252928 9804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.67
I20250625 07:58:47.288398 9804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:37239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.67:45551
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.85.67
--webserver_port=36077
--tserver_master_addrs=127.5.85.126:43881
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.67
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:58:47.289729 9804 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:58:47.291457 9804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:58:47.307722 9821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:58:48.175801 9826 raft_consensus.cc:491] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:58:48.176259 9826 raft_consensus.cc:513] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:48.177774 9826 leader_election.cc:290] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573), 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551)
W20250625 07:58:48.187019 9540 leader_election.cc:336] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551): Network error: Client connection negotiation failed: client connection to 127.5.85.67:45551: connect: Connection refused (error 111)
I20250625 07:58:48.193037 9754 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "557364b8ffd8492f8f2a2a616a7c127e" candidate_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "cc8ab07892f84f6f910e5ee36cab6300" is_pre_election: true
I20250625 07:58:48.193694 9754 raft_consensus.cc:2466] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9b866b17f3446a6bd5e380d224c6aeb in term 2.
I20250625 07:58:48.195080 9543 leader_election.cc:304] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc8ab07892f84f6f910e5ee36cab6300, f9b866b17f3446a6bd5e380d224c6aeb; no voters: 92b13ef756c54976a43d1e79ab68a602
I20250625 07:58:48.195844 9826 raft_consensus.cc:2802] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250625 07:58:48.196134 9826 raft_consensus.cc:491] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:58:48.196377 9826 raft_consensus.cc:3058] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Advancing to term 3
I20250625 07:58:48.202708 9826 raft_consensus.cc:513] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:48.204160 9826 leader_election.cc:290] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 election: Requested vote from peers cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573), 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551)
I20250625 07:58:48.205164 9754 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "557364b8ffd8492f8f2a2a616a7c127e" candidate_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 13 } } ignore_live_leader: false dest_uuid: "cc8ab07892f84f6f910e5ee36cab6300"
I20250625 07:58:48.205721 9754 raft_consensus.cc:3058] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Advancing to term 3
W20250625 07:58:48.207515 9540 leader_election.cc:336] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551): Network error: Client connection negotiation failed: client connection to 127.5.85.67:45551: connect: Connection refused (error 111)
I20250625 07:58:48.212363 9754 raft_consensus.cc:2466] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b866b17f3446a6bd5e380d224c6aeb in term 3.
I20250625 07:58:48.213492 9543 leader_election.cc:304] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc8ab07892f84f6f910e5ee36cab6300, f9b866b17f3446a6bd5e380d224c6aeb; no voters: 92b13ef756c54976a43d1e79ab68a602
I20250625 07:58:48.214286 9826 raft_consensus.cc:2802] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250625 07:58:48.215865 9826 raft_consensus.cc:695] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 3 LEADER]: Becoming Leader. State: Replica: f9b866b17f3446a6bd5e380d224c6aeb, State: Running, Role: LEADER
I20250625 07:58:48.216614 9826 consensus_queue.cc:237] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:48.226123 9466 catalog_manager.cc:5582] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb reported cstate change: term changed from 2 to 3, leader changed from <none> to f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65). New cstate: current_term: 3 leader_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250625 07:58:47.307735 9820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:47.308233 9823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:58:48.393306 9822 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:58:48.393407 9804 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:58:48.397279 9804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:58:48.400036 9804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:58:48.401511 9804 hybrid_clock.cc:648] HybridClock initialized: now 1750838328401471 us; error 53 us; skew 500 ppm
I20250625 07:58:48.402370 9804 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:58:48.409109 9804 webserver.cc:469] Webserver started at http://127.5.85.67:36077/ using document root <none> and password file <none>
I20250625 07:58:48.410060 9804 fs_manager.cc:362] Metadata directory not provided
I20250625 07:58:48.410297 9804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:58:48.418490 9804 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250625 07:58:48.423856 9841 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:58:48.425350 9804 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.005s
I20250625 07:58:48.425732 9804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "92b13ef756c54976a43d1e79ab68a602"
format_stamp: "Formatted at 2025-06-25 07:58:20 on dist-test-slave-8zlz"
I20250625 07:58:48.428035 9804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:58:48.483119 9804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:58:48.484548 9804 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:58:48.485013 9804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:58:48.487578 9804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:58:48.493250 9848 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250625 07:58:48.505059 9804 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250625 07:58:48.505339 9804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250625 07:58:48.505620 9804 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250625 07:58:48.511746 9848 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Bootstrap starting.
I20250625 07:58:48.514323 9804 ts_tablet_manager.cc:610] Registered 2 tablets
I20250625 07:58:48.514530 9804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.002s sys 0.003s
I20250625 07:58:48.520187 9826 raft_consensus.cc:491] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:58:48.520666 9826 raft_consensus.cc:513] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:48.522564 9826 leader_election.cc:290] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551), cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573)
I20250625 07:58:48.524112 9754 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f98cc844fb24ece82373c7f157fa07c" candidate_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "cc8ab07892f84f6f910e5ee36cab6300" is_pre_election: true
I20250625 07:58:48.524719 9754 raft_consensus.cc:2466] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f9b866b17f3446a6bd5e380d224c6aeb in term 2.
I20250625 07:58:48.525942 9543 leader_election.cc:304] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc8ab07892f84f6f910e5ee36cab6300, f9b866b17f3446a6bd5e380d224c6aeb; no voters:
I20250625 07:58:48.526880 9826 raft_consensus.cc:2802] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250625 07:58:48.527215 9826 raft_consensus.cc:491] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250625 07:58:48.527247 9540 leader_election.cc:336] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551): Network error: Client connection negotiation failed: client connection to 127.5.85.67:45551: connect: Connection refused (error 111)
I20250625 07:58:48.527559 9826 raft_consensus.cc:3058] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 2 FOLLOWER]: Advancing to term 3
I20250625 07:58:48.534072 9826 raft_consensus.cc:513] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:48.537142 9826 leader_election.cc:290] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 election: Requested vote from peers 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551), cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573)
I20250625 07:58:48.538944 9754 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5f98cc844fb24ece82373c7f157fa07c" candidate_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "cc8ab07892f84f6f910e5ee36cab6300"
I20250625 07:58:48.539505 9754 raft_consensus.cc:3058] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 2 FOLLOWER]: Advancing to term 3
W20250625 07:58:48.542114 9540 leader_election.cc:336] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551): Network error: Client connection negotiation failed: client connection to 127.5.85.67:45551: connect: Connection refused (error 111)
I20250625 07:58:48.545698 9754 raft_consensus.cc:2466] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f9b866b17f3446a6bd5e380d224c6aeb in term 3.
I20250625 07:58:48.546986 9543 leader_election.cc:304] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: cc8ab07892f84f6f910e5ee36cab6300, f9b866b17f3446a6bd5e380d224c6aeb; no voters: 92b13ef756c54976a43d1e79ab68a602
I20250625 07:58:48.547959 9826 raft_consensus.cc:2802] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 3 FOLLOWER]: Leader election won for term 3
I20250625 07:58:48.548388 9826 raft_consensus.cc:695] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [term 3 LEADER]: Becoming Leader. State: Replica: f9b866b17f3446a6bd5e380d224c6aeb, State: Running, Role: LEADER
I20250625 07:58:48.549135 9826 consensus_queue.cc:237] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:48.558990 9466 catalog_manager.cc:5582] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb reported cstate change: term changed from 2 to 3, leader changed from 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) to f9b866b17f3446a6bd5e380d224c6aeb (127.5.85.65). New cstate: current_term: 3 leader_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250625 07:58:48.577395 9848 log.cc:826] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Log is configured to *not* fsync() on all Append() calls
I20250625 07:58:48.684999 9804 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.67:45551
I20250625 07:58:48.685220 9956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.67:45551 every 8 connection(s)
I20250625 07:58:48.685426 9848 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=350 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:48.686270 9848 tablet_bootstrap.cc:492] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Bootstrap complete.
I20250625 07:58:48.687431 9848 ts_tablet_manager.cc:1397] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Time spent bootstrapping tablet: real 0.176s user 0.143s sys 0.029s
I20250625 07:58:48.687796 9804 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 07:58:48.694527 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 9804
I20250625 07:58:48.710606 9754 raft_consensus.cc:1273] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 3 FOLLOWER]: Refusing update from remote peer f9b866b17f3446a6bd5e380d224c6aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250625 07:58:48.712541 9826 consensus_queue.cc:1035] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250625 07:58:48.712322 9848 raft_consensus.cc:357] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:48.715400 9848 raft_consensus.cc:738] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b13ef756c54976a43d1e79ab68a602, State: Initialized, Role: FOLLOWER
I20250625 07:58:48.716475 9848 consensus_queue.cc:260] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 2.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false } }
I20250625 07:58:48.726315 9848 ts_tablet_manager.cc:1428] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Time spent starting tablet: real 0.039s user 0.029s sys 0.004s
I20250625 07:58:48.727161 9848 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Bootstrap starting.
I20250625 07:58:48.729033 9957 heartbeater.cc:344] Connected to a master server at 127.5.85.126:43881
I20250625 07:58:48.729455 9957 heartbeater.cc:461] Registering TS with master...
I20250625 07:58:48.730782 9957 heartbeater.cc:507] Master 127.5.85.126:43881 requested a full tablet report, sending...
I20250625 07:58:48.735491 9466 ts_manager.cc:194] Registered new tserver with Master: 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551)
I20250625 07:58:48.739681 9466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.67:51845
I20250625 07:58:48.744392 9957 heartbeater.cc:499] Master 127.5.85.126:43881 was elected leader, sending a full tablet report...
I20250625 07:58:48.749107 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:58:48.756500 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:58:48.757639 9912 raft_consensus.cc:3058] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 2 FOLLOWER]: Advancing to term 3
I20250625 07:58:48.762567 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f9b866b17f3446a6bd5e380d224c6aeb to finish bootstrapping
I20250625 07:58:48.769071 9912 raft_consensus.cc:1273] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 3 FOLLOWER]: Refusing update from remote peer f9b866b17f3446a6bd5e380d224c6aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250625 07:58:48.772439 9826 consensus_queue.cc:1035] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20250625 07:58:48.797811 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cc8ab07892f84f6f910e5ee36cab6300 to finish bootstrapping
I20250625 07:58:48.816481 9609 consensus_queue.cc:237] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 13, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:48.820649 9754 raft_consensus.cc:1273] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 3 FOLLOWER]: Refusing update from remote peer f9b866b17f3446a6bd5e380d224c6aeb: Log matching property violated. Preceding OpId in replica: term: 3 index: 14. Preceding OpId from leader: term: 3 index: 15. (index mismatch)
I20250625 07:58:48.821758 9826 consensus_queue.cc:1035] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20250625 07:58:48.823421 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 92b13ef756c54976a43d1e79ab68a602 to finish bootstrapping
I20250625 07:58:48.827730 9831 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } }
I20250625 07:58:48.829358 9754 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 3 FOLLOWER]: Committing config change with OpId 3.15: config changed from index 13 to 15, VOTER 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } }
I20250625 07:58:48.840202 9454 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 557364b8ffd8492f8f2a2a616a7c127e with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 07:58:48.842350 9467 catalog_manager.cc:5582] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 reported cstate change: config changed from index 13 to 15, VOTER 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67) evicted. New cstate: current_term: 3 leader_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } } }
I20250625 07:58:48.852456 9609 consensus_queue.cc:237] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 3.15, Last appended by leader: 13, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } }
I20250625 07:58:48.857491 9831 raft_consensus.cc:2953] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb [term 3 LEADER]: Committing config change with OpId 3.16: config changed from index 15 to 16, VOTER cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) evicted. New config: { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } } }
I20250625 07:58:48.868947 9454 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 557364b8ffd8492f8f2a2a616a7c127e with cas_config_opid_index 15: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 07:58:48.865587 9892 tablet_service.cc:1515] Processing DeleteTablet for tablet 557364b8ffd8492f8f2a2a616a7c127e with delete_type TABLET_DATA_TOMBSTONED (TS 92b13ef756c54976a43d1e79ab68a602 not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:49386
I20250625 07:58:48.872205 9976 tablet_replica.cc:331] stopping tablet replica
I20250625 07:58:48.872987 9976 raft_consensus.cc:2241] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250625 07:58:48.873550 9976 raft_consensus.cc:2270] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250625 07:58:48.875869 9467 catalog_manager.cc:5582] T 557364b8ffd8492f8f2a2a616a7c127e P f9b866b17f3446a6bd5e380d224c6aeb reported cstate change: config changed from index 15 to 16, VOTER cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66) evicted. New cstate: current_term: 3 leader_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" committed_config { opid_index: 16 OBSOLETE_local: true peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } health_report { overall_health: HEALTHY } } }
I20250625 07:58:48.878122 9976 ts_tablet_manager.cc:1905] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250625 07:58:48.880290 9848 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:58:48.881167 9848 tablet_bootstrap.cc:492] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Bootstrap complete.
I20250625 07:58:48.882560 9848 ts_tablet_manager.cc:1397] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Time spent bootstrapping tablet: real 0.156s user 0.114s sys 0.013s
I20250625 07:58:48.888154 9848 raft_consensus.cc:357] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:48.888793 9848 raft_consensus.cc:738] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b13ef756c54976a43d1e79ab68a602, State: Initialized, Role: FOLLOWER
I20250625 07:58:48.889473 9848 consensus_queue.cc:260] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 } } peers { permanent_uuid: "f9b866b17f3446a6bd5e380d224c6aeb" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 43657 } attrs { promote: false } } peers { permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false } }
I20250625 07:58:48.891294 9848 ts_tablet_manager.cc:1428] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602: Time spent starting tablet: real 0.008s user 0.004s sys 0.000s
I20250625 07:58:48.893283 9976 ts_tablet_manager.cc:1918] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20250625 07:58:48.893700 9976 log.cc:1199] T 557364b8ffd8492f8f2a2a616a7c127e P 92b13ef756c54976a43d1e79ab68a602: Deleting WAL directory at /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/wals/557364b8ffd8492f8f2a2a616a7c127e
I20250625 07:58:48.895504 9451 catalog_manager.cc:4928] TS 92b13ef756c54976a43d1e79ab68a602 (127.5.85.67:45551): tablet 557364b8ffd8492f8f2a2a616a7c127e (table TestTable [id=4c0cf8a8a0074dd2a4587163f74361ce]) successfully deleted
I20250625 07:58:48.896682 9734 tablet_service.cc:1515] Processing DeleteTablet for tablet 557364b8ffd8492f8f2a2a616a7c127e with delete_type TABLET_DATA_TOMBSTONED (TS cc8ab07892f84f6f910e5ee36cab6300 not found in new config with opid_index 16) from {username='slave'} at 127.0.0.1:49320
I20250625 07:58:48.899503 9977 tablet_replica.cc:331] stopping tablet replica
I20250625 07:58:48.900310 9977 raft_consensus.cc:2241] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250625 07:58:48.900792 9977 raft_consensus.cc:2270] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250625 07:58:48.903812 9977 ts_tablet_manager.cc:1905] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250625 07:58:48.915280 9977 ts_tablet_manager.cc:1918] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.15
I20250625 07:58:48.915638 9977 log.cc:1199] T 557364b8ffd8492f8f2a2a616a7c127e P cc8ab07892f84f6f910e5ee36cab6300: Deleting WAL directory at /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/wals/557364b8ffd8492f8f2a2a616a7c127e
I20250625 07:58:48.917155 9454 catalog_manager.cc:4928] TS cc8ab07892f84f6f910e5ee36cab6300 (127.5.85.66:45573): tablet 557364b8ffd8492f8f2a2a616a7c127e (table TestTable [id=4c0cf8a8a0074dd2a4587163f74361ce]) successfully deleted
I20250625 07:58:49.078470 9754 raft_consensus.cc:1273] T 5f98cc844fb24ece82373c7f157fa07c P cc8ab07892f84f6f910e5ee36cab6300 [term 3 FOLLOWER]: Refusing update from remote peer f9b866b17f3446a6bd5e380d224c6aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250625 07:58:49.079622 9831 consensus_queue.cc:1035] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc8ab07892f84f6f910e5ee36cab6300" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 45573 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250625 07:58:49.087483 9912 raft_consensus.cc:3058] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 2 FOLLOWER]: Advancing to term 3
I20250625 07:58:49.093547 9912 raft_consensus.cc:1273] T 5f98cc844fb24ece82373c7f157fa07c P 92b13ef756c54976a43d1e79ab68a602 [term 3 FOLLOWER]: Refusing update from remote peer f9b866b17f3446a6bd5e380d224c6aeb: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250625 07:58:49.094821 9979 consensus_queue.cc:1035] T 5f98cc844fb24ece82373c7f157fa07c P f9b866b17f3446a6bd5e380d224c6aeb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b13ef756c54976a43d1e79ab68a602" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 45551 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20250625 07:59:03.739050 9650 debug-util.cc:398] Leaking SignalData structure 0x7b08000c4360 after lost signal to thread 9524
W20250625 07:59:03.740033 9650 debug-util.cc:398] Leaking SignalData structure 0x7b08000b44e0 after lost signal to thread 9653
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Timed out: status_and_schema {
tablet_status {
tablet_id: "5f98cc844fb24ece82373c7f157fa07c"
table_name: "TestTable1"
state: RUNNING
last_status: "Bootstrap complete."
estimated_on_disk_size: 8415792
tablet_data_state: TABLET_DATA_READY
partition {
partition_key_start: ""
partition_key_end: ""
}
data_dirs: "/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/data"
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: FOLLOWER
}
status_and_schema {
tablet_status {
tablet_id: "557364b8ffd8492f8f2a2a616a7c127e"
table_name: "TestTable"
state: STOPPED
last_status: "Deleted tablet blocks from disk"
estimated_on_disk_size: 8401336
tablet_data_state: TABLET_DATA_TOMBSTONED
partition {
partition_key_start: ""
partition_key_end: ""
}
}
schema {
columns {
id: 10
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 11
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
id: 12
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
partition_schema {
range_schema {
columns {
id: 10
}
}
}
schema_version: 0
role: UNKNOWN_ROLE
}
I20250625 07:59:08.846904 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 9504
I20250625 07:59:08.873809 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 9657
I20250625 07:59:08.899289 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 9804
I20250625 07:59:08.922704 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 9433
2025-06-25T07:59:08Z chronyd exiting
I20250625 07:59:08.966778 5461 test_util.cc:183] -----------------------------------------------
I20250625 07:59:08.966984 5461 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1750838232495789-5461-0
[ FAILED ] AdminCliTest.TestRebuildTables (54912 ms)
[----------] 5 tests from AdminCliTest (116401 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250625 07:59:08.970359 5461 test_util.cc:276] Using random seed: -774276769
I20250625 07:59:08.974591 5461 ts_itest-base.cc:115] Starting cluster with:
I20250625 07:59:08.974805 5461 ts_itest-base.cc:116] --------------
I20250625 07:59:08.974928 5461 ts_itest-base.cc:117] 5 tablet servers
I20250625 07:59:08.975029 5461 ts_itest-base.cc:118] 3 replicas per TS
I20250625 07:59:08.975131 5461 ts_itest-base.cc:119] --------------
2025-06-25T07:59:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T07:59:08Z Disabled control of system clock
I20250625 07:59:09.016039 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:44819
--webserver_interface=127.5.85.126
--webserver_port=0
--builtin_ntp_servers=127.5.85.84:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:44819
--raft_prepare_replacement_before_eviction=true with env {}
W20250625 07:59:09.313661 9999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:09.314232 9999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:09.314667 9999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:09.346472 9999 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 07:59:09.346875 9999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:59:09.347085 9999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:09.347282 9999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:59:09.347479 9999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:59:09.383142 9999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:44819
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:44819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:09.384428 9999 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:09.386044 9999 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:09.400143 10006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:09.400569 10005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:09.401203 10008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:09.401917 9999 server_base.cc:1048] running on GCE node
I20250625 07:59:10.500281 9999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:10.503324 9999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:10.504801 9999 hybrid_clock.cc:648] HybridClock initialized: now 1750838350504753 us; error 64 us; skew 500 ppm
I20250625 07:59:10.505595 9999 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:10.512241 9999 webserver.cc:469] Webserver started at http://127.5.85.126:40357/ using document root <none> and password file <none>
I20250625 07:59:10.513201 9999 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:10.513401 9999 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:10.513842 9999 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:10.518396 9999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8c6c069ba28843c28e12b677d523a36b"
format_stamp: "Formatted at 2025-06-25 07:59:10 on dist-test-slave-8zlz"
I20250625 07:59:10.519585 9999 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8c6c069ba28843c28e12b677d523a36b"
format_stamp: "Formatted at 2025-06-25 07:59:10 on dist-test-slave-8zlz"
I20250625 07:59:10.527185 9999 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250625 07:59:10.532905 10015 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:10.534123 9999 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250625 07:59:10.534435 9999 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8c6c069ba28843c28e12b677d523a36b"
format_stamp: "Formatted at 2025-06-25 07:59:10 on dist-test-slave-8zlz"
I20250625 07:59:10.534814 9999 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:10.588698 9999 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:10.590283 9999 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:10.590731 9999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:10.657744 9999 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:44819
I20250625 07:59:10.657801 10066 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:44819 every 8 connection(s)
I20250625 07:59:10.660358 9999 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 07:59:10.665191 10067 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:10.670765 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 9999
I20250625 07:59:10.671167 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 07:59:10.686173 10067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b: Bootstrap starting.
I20250625 07:59:10.692714 10067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:10.694927 10067 log.cc:826] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:10.699437 10067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b: No bootstrap required, opened a new log
I20250625 07:59:10.717386 10067 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c069ba28843c28e12b677d523a36b" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 44819 } }
I20250625 07:59:10.718336 10067 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:10.718556 10067 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6c069ba28843c28e12b677d523a36b, State: Initialized, Role: FOLLOWER
I20250625 07:59:10.719247 10067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c069ba28843c28e12b677d523a36b" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 44819 } }
I20250625 07:59:10.719727 10067 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:59:10.719978 10067 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:59:10.720273 10067 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:10.724568 10067 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c069ba28843c28e12b677d523a36b" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 44819 } }
I20250625 07:59:10.725423 10067 leader_election.cc:304] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c6c069ba28843c28e12b677d523a36b; no voters:
I20250625 07:59:10.727043 10067 leader_election.cc:290] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:59:10.727725 10072 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:59:10.729585 10072 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [term 1 LEADER]: Becoming Leader. State: Replica: 8c6c069ba28843c28e12b677d523a36b, State: Running, Role: LEADER
I20250625 07:59:10.730302 10072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c069ba28843c28e12b677d523a36b" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 44819 } }
I20250625 07:59:10.731448 10067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:59:10.739951 10074 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c6c069ba28843c28e12b677d523a36b. Latest consensus state: current_term: 1 leader_uuid: "8c6c069ba28843c28e12b677d523a36b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c069ba28843c28e12b677d523a36b" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 44819 } } }
I20250625 07:59:10.741025 10073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c6c069ba28843c28e12b677d523a36b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6c069ba28843c28e12b677d523a36b" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 44819 } } }
I20250625 07:59:10.741482 10074 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [sys.catalog]: This master's current role is: LEADER
I20250625 07:59:10.741523 10073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b [sys.catalog]: This master's current role is: LEADER
I20250625 07:59:10.744053 10082 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:59:10.753625 10082 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:59:10.769306 10082 catalog_manager.cc:1349] Generated new cluster ID: 3528238666c84959816abd298c40ef88
I20250625 07:59:10.769560 10082 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:59:10.790397 10082 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 07:59:10.791867 10082 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:59:10.807905 10082 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8c6c069ba28843c28e12b677d523a36b: Generated new TSK 0
I20250625 07:59:10.808790 10082 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 07:59:10.820255 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:0
--local_ip_for_outbound_sockets=127.5.85.65
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:44819
--builtin_ntp_servers=127.5.85.84:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250625 07:59:11.141469 10091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:11.141988 10091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:11.142468 10091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:11.173739 10091 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 07:59:11.174163 10091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:11.174978 10091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:59:11.211114 10091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:44819
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:11.212399 10091 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:11.214042 10091 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:11.231343 10098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:11.233295 10097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:11.232374 10100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:12.329924 10099 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:59:12.330041 10091 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:59:12.333716 10091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:12.336325 10091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:12.337750 10091 hybrid_clock.cc:648] HybridClock initialized: now 1750838352337675 us; error 104 us; skew 500 ppm
I20250625 07:59:12.338557 10091 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:12.345264 10091 webserver.cc:469] Webserver started at http://127.5.85.65:44035/ using document root <none> and password file <none>
I20250625 07:59:12.346230 10091 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:12.346434 10091 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:12.347009 10091 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:12.351507 10091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "962dd68c965349f7aa10753df3effe23"
format_stamp: "Formatted at 2025-06-25 07:59:12 on dist-test-slave-8zlz"
I20250625 07:59:12.352676 10091 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "962dd68c965349f7aa10753df3effe23"
format_stamp: "Formatted at 2025-06-25 07:59:12 on dist-test-slave-8zlz"
I20250625 07:59:12.360263 10091 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20250625 07:59:12.366325 10107 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:12.367543 10091 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 07:59:12.367863 10091 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "962dd68c965349f7aa10753df3effe23"
format_stamp: "Formatted at 2025-06-25 07:59:12 on dist-test-slave-8zlz"
I20250625 07:59:12.368183 10091 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:12.440775 10091 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:12.442225 10091 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:12.442682 10091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:12.445518 10091 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:12.449666 10091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:59:12.449875 10091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:12.450119 10091 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:59:12.450268 10091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:12.591078 10091 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:40319
I20250625 07:59:12.591192 10219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:40319 every 8 connection(s)
I20250625 07:59:12.593641 10091 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 07:59:12.602815 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 10091
I20250625 07:59:12.603226 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 07:59:12.609763 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.66:0
--local_ip_for_outbound_sockets=127.5.85.66
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:44819
--builtin_ntp_servers=127.5.85.84:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 07:59:12.615288 10220 heartbeater.cc:344] Connected to a master server at 127.5.85.126:44819
I20250625 07:59:12.615801 10220 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:12.616854 10220 heartbeater.cc:507] Master 127.5.85.126:44819 requested a full tablet report, sending...
I20250625 07:59:12.619477 10032 ts_manager.cc:194] Registered new tserver with Master: 962dd68c965349f7aa10753df3effe23 (127.5.85.65:40319)
I20250625 07:59:12.621502 10032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:48651
W20250625 07:59:12.902558 10224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:12.903105 10224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:12.903652 10224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:12.935158 10224 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 07:59:12.935540 10224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:12.936311 10224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.66
I20250625 07:59:12.971002 10224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:44819
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.66
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:12.972277 10224 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:12.973839 10224 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:12.988947 10231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:13.625192 10220 heartbeater.cc:499] Master 127.5.85.126:44819 was elected leader, sending a full tablet report...
W20250625 07:59:12.989143 10234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:12.991174 10224 server_base.cc:1048] running on GCE node
W20250625 07:59:12.988969 10232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:14.069334 10224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:14.072018 10224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:14.073491 10224 hybrid_clock.cc:648] HybridClock initialized: now 1750838354073424 us; error 80 us; skew 500 ppm
I20250625 07:59:14.074369 10224 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:14.080807 10224 webserver.cc:469] Webserver started at http://127.5.85.66:38327/ using document root <none> and password file <none>
I20250625 07:59:14.081702 10224 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:14.081926 10224 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:14.082381 10224 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:14.088353 10224 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cc5d0afaccda48de8107086e61007220"
format_stamp: "Formatted at 2025-06-25 07:59:14 on dist-test-slave-8zlz"
I20250625 07:59:14.089453 10224 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cc5d0afaccda48de8107086e61007220"
format_stamp: "Formatted at 2025-06-25 07:59:14 on dist-test-slave-8zlz"
I20250625 07:59:14.096354 10224 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250625 07:59:14.102114 10241 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:14.103189 10224 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 07:59:14.103513 10224 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cc5d0afaccda48de8107086e61007220"
format_stamp: "Formatted at 2025-06-25 07:59:14 on dist-test-slave-8zlz"
I20250625 07:59:14.103849 10224 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:14.155898 10224 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:14.157315 10224 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:14.157718 10224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:14.160199 10224 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:14.164204 10224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:59:14.164415 10224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:14.164651 10224 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:59:14.164808 10224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:14.287671 10224 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.66:33927
I20250625 07:59:14.287765 10353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.66:33927 every 8 connection(s)
I20250625 07:59:14.290140 10224 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250625 07:59:14.293460 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 10224
I20250625 07:59:14.294178 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250625 07:59:14.301915 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.67:0
--local_ip_for_outbound_sockets=127.5.85.67
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:44819
--builtin_ntp_servers=127.5.85.84:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 07:59:14.311486 10354 heartbeater.cc:344] Connected to a master server at 127.5.85.126:44819
I20250625 07:59:14.311997 10354 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:14.313241 10354 heartbeater.cc:507] Master 127.5.85.126:44819 requested a full tablet report, sending...
I20250625 07:59:14.315632 10032 ts_manager.cc:194] Registered new tserver with Master: cc5d0afaccda48de8107086e61007220 (127.5.85.66:33927)
I20250625 07:59:14.316901 10032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.66:40753
W20250625 07:59:14.599911 10358 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:14.600457 10358 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:14.600993 10358 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:14.633589 10358 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 07:59:14.634007 10358 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:14.634845 10358 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.67
I20250625 07:59:14.669788 10358 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:44819
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.67
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:14.671139 10358 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:14.672811 10358 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:14.689532 10365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:15.320557 10354 heartbeater.cc:499] Master 127.5.85.126:44819 was elected leader, sending a full tablet report...
W20250625 07:59:14.689745 10367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:14.689525 10364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:15.795044 10366 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:59:15.795176 10358 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:59:15.798949 10358 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:15.801270 10358 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:15.802682 10358 hybrid_clock.cc:648] HybridClock initialized: now 1750838355802606 us; error 56 us; skew 500 ppm
I20250625 07:59:15.803563 10358 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:15.810165 10358 webserver.cc:469] Webserver started at http://127.5.85.67:38161/ using document root <none> and password file <none>
I20250625 07:59:15.811199 10358 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:15.811425 10358 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:15.811893 10358 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:15.816377 10358 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5b289efe11e547c888b108bee7fbb8b1"
format_stamp: "Formatted at 2025-06-25 07:59:15 on dist-test-slave-8zlz"
I20250625 07:59:15.817524 10358 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5b289efe11e547c888b108bee7fbb8b1"
format_stamp: "Formatted at 2025-06-25 07:59:15 on dist-test-slave-8zlz"
I20250625 07:59:15.824739 10358 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250625 07:59:15.830444 10374 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:15.831501 10358 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 07:59:15.831813 10358 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5b289efe11e547c888b108bee7fbb8b1"
format_stamp: "Formatted at 2025-06-25 07:59:15 on dist-test-slave-8zlz"
I20250625 07:59:15.832140 10358 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:15.882560 10358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:15.884102 10358 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:15.884558 10358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:15.887120 10358 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:15.891424 10358 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:59:15.891702 10358 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:15.891963 10358 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:59:15.892139 10358 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:16.024430 10358 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.67:39109
I20250625 07:59:16.024498 10486 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.67:39109 every 8 connection(s)
I20250625 07:59:16.027895 10358 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250625 07:59:16.029549 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 10358
I20250625 07:59:16.030085 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250625 07:59:16.040872 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.68:0
--local_ip_for_outbound_sockets=127.5.85.68
--webserver_interface=127.5.85.68
--webserver_port=0
--tserver_master_addrs=127.5.85.126:44819
--builtin_ntp_servers=127.5.85.84:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 07:59:16.055430 10487 heartbeater.cc:344] Connected to a master server at 127.5.85.126:44819
I20250625 07:59:16.056022 10487 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:16.057420 10487 heartbeater.cc:507] Master 127.5.85.126:44819 requested a full tablet report, sending...
I20250625 07:59:16.059911 10032 ts_manager.cc:194] Registered new tserver with Master: 5b289efe11e547c888b108bee7fbb8b1 (127.5.85.67:39109)
I20250625 07:59:16.061579 10032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.67:50961
W20250625 07:59:16.344157 10491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:16.344664 10491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:16.345160 10491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:16.376852 10491 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 07:59:16.377261 10491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:16.378033 10491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.68
I20250625 07:59:16.412410 10491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.5.85.68
--webserver_port=0
--tserver_master_addrs=127.5.85.126:44819
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.68
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:16.413683 10491 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:16.415364 10491 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:16.432569 10500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:17.064983 10487 heartbeater.cc:499] Master 127.5.85.126:44819 was elected leader, sending a full tablet report...
W20250625 07:59:16.432580 10497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:16.432569 10498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:17.520836 10499 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:59:17.520954 10491 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:59:17.525213 10491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:17.527490 10491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:17.528924 10491 hybrid_clock.cc:648] HybridClock initialized: now 1750838357528884 us; error 48 us; skew 500 ppm
I20250625 07:59:17.529722 10491 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:17.541314 10491 webserver.cc:469] Webserver started at http://127.5.85.68:36605/ using document root <none> and password file <none>
I20250625 07:59:17.542272 10491 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:17.542502 10491 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:17.542982 10491 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:17.547531 10491 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "d21b5df4ddaf48cdbe4ff12cc50f5122"
format_stamp: "Formatted at 2025-06-25 07:59:17 on dist-test-slave-8zlz"
I20250625 07:59:17.548671 10491 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "d21b5df4ddaf48cdbe4ff12cc50f5122"
format_stamp: "Formatted at 2025-06-25 07:59:17 on dist-test-slave-8zlz"
I20250625 07:59:17.557806 10491 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.001s sys 0.008s
I20250625 07:59:17.563495 10507 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:17.564642 10491 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250625 07:59:17.565076 10491 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "d21b5df4ddaf48cdbe4ff12cc50f5122"
format_stamp: "Formatted at 2025-06-25 07:59:17 on dist-test-slave-8zlz"
I20250625 07:59:17.565457 10491 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:17.616145 10491 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:17.617607 10491 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:17.618046 10491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:17.620524 10491 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:17.624696 10491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:59:17.624936 10491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:17.625205 10491 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:59:17.625355 10491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:17.757316 10491 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.68:45557
I20250625 07:59:17.757426 10619 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.68:45557 every 8 connection(s)
I20250625 07:59:17.759904 10491 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250625 07:59:17.770540 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 10491
I20250625 07:59:17.771075 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250625 07:59:17.777575 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.69:0
--local_ip_for_outbound_sockets=127.5.85.69
--webserver_interface=127.5.85.69
--webserver_port=0
--tserver_master_addrs=127.5.85.126:44819
--builtin_ntp_servers=127.5.85.84:38365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250625 07:59:17.782022 10620 heartbeater.cc:344] Connected to a master server at 127.5.85.126:44819
I20250625 07:59:17.782569 10620 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:17.783922 10620 heartbeater.cc:507] Master 127.5.85.126:44819 requested a full tablet report, sending...
I20250625 07:59:17.786322 10032 ts_manager.cc:194] Registered new tserver with Master: d21b5df4ddaf48cdbe4ff12cc50f5122 (127.5.85.68:45557)
I20250625 07:59:17.787757 10032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.68:40317
W20250625 07:59:18.076180 10624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:18.076687 10624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:18.077198 10624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:18.109825 10624 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250625 07:59:18.110251 10624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:18.111095 10624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.69
I20250625 07:59:18.146571 10624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:38365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.5.85.69
--webserver_port=0
--tserver_master_addrs=127.5.85.126:44819
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.69
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:18.147912 10624 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:18.149575 10624 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:18.166491 10630 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:18.791129 10620 heartbeater.cc:499] Master 127.5.85.126:44819 was elected leader, sending a full tablet report...
W20250625 07:59:18.168491 10631 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:18.168553 10624 server_base.cc:1048] running on GCE node
W20250625 07:59:18.166558 10633 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:19.256026 10624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:19.258426 10624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:19.259821 10624 hybrid_clock.cc:648] HybridClock initialized: now 1750838359259772 us; error 71 us; skew 500 ppm
I20250625 07:59:19.260643 10624 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:19.267550 10624 webserver.cc:469] Webserver started at http://127.5.85.69:43425/ using document root <none> and password file <none>
I20250625 07:59:19.268514 10624 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:19.268744 10624 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:19.269212 10624 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:19.275790 10624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "465b6db295aa4ea387b3a1112519672b"
format_stamp: "Formatted at 2025-06-25 07:59:19 on dist-test-slave-8zlz"
I20250625 07:59:19.276976 10624 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "465b6db295aa4ea387b3a1112519672b"
format_stamp: "Formatted at 2025-06-25 07:59:19 on dist-test-slave-8zlz"
I20250625 07:59:19.284348 10624 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250625 07:59:19.291317 10640 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:19.292768 10624 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20250625 07:59:19.293255 10624 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "465b6db295aa4ea387b3a1112519672b"
format_stamp: "Formatted at 2025-06-25 07:59:19 on dist-test-slave-8zlz"
I20250625 07:59:19.293722 10624 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:19.346851 10624 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:19.348340 10624 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:19.348773 10624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:19.351336 10624 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:19.355507 10624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:59:19.355806 10624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:19.356073 10624 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:59:19.356220 10624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:19.493011 10624 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.69:44751
I20250625 07:59:19.493088 10753 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.69:44751 every 8 connection(s)
I20250625 07:59:19.495435 10624 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250625 07:59:19.496829 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 10624
I20250625 07:59:19.497225 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250625 07:59:19.518030 10754 heartbeater.cc:344] Connected to a master server at 127.5.85.126:44819
I20250625 07:59:19.518549 10754 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:19.519644 10754 heartbeater.cc:507] Master 127.5.85.126:44819 requested a full tablet report, sending...
I20250625 07:59:19.521544 10032 ts_manager.cc:194] Registered new tserver with Master: 465b6db295aa4ea387b3a1112519672b (127.5.85.69:44751)
I20250625 07:59:19.522855 10032 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.69:34259
I20250625 07:59:19.533911 5461 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250625 07:59:19.577231 10032 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51220:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250625 07:59:19.658147 10689 tablet_service.cc:1468] Processing CreateTablet for tablet f410c5630e934badab29f22d56a5115f (DEFAULT_TABLE table=TestTable [id=4e6aa07a5fad4ddfac7d58ba49090b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:59:19.660394 10689 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f410c5630e934badab29f22d56a5115f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:19.659714 10289 tablet_service.cc:1468] Processing CreateTablet for tablet f410c5630e934badab29f22d56a5115f (DEFAULT_TABLE table=TestTable [id=4e6aa07a5fad4ddfac7d58ba49090b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:59:19.661597 10289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f410c5630e934badab29f22d56a5115f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:19.670760 10422 tablet_service.cc:1468] Processing CreateTablet for tablet f410c5630e934badab29f22d56a5115f (DEFAULT_TABLE table=TestTable [id=4e6aa07a5fad4ddfac7d58ba49090b41]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:59:19.672919 10422 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f410c5630e934badab29f22d56a5115f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:19.696810 10773 tablet_bootstrap.cc:492] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b: Bootstrap starting.
I20250625 07:59:19.708797 10775 tablet_bootstrap.cc:492] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1: Bootstrap starting.
I20250625 07:59:19.708997 10773 tablet_bootstrap.cc:654] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:19.710455 10774 tablet_bootstrap.cc:492] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220: Bootstrap starting.
I20250625 07:59:19.713749 10773 log.cc:826] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:19.719436 10774 tablet_bootstrap.cc:654] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:19.720305 10775 tablet_bootstrap.cc:654] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:19.722887 10774 log.cc:826] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:19.723162 10775 log.cc:826] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:19.731134 10773 tablet_bootstrap.cc:492] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b: No bootstrap required, opened a new log
I20250625 07:59:19.731818 10773 ts_tablet_manager.cc:1397] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b: Time spent bootstrapping tablet: real 0.036s user 0.017s sys 0.013s
I20250625 07:59:19.733330 10774 tablet_bootstrap.cc:492] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220: No bootstrap required, opened a new log
I20250625 07:59:19.734268 10774 ts_tablet_manager.cc:1397] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.006s
I20250625 07:59:19.734830 10775 tablet_bootstrap.cc:492] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1: No bootstrap required, opened a new log
I20250625 07:59:19.735579 10775 ts_tablet_manager.cc:1397] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1: Time spent bootstrapping tablet: real 0.028s user 0.010s sys 0.012s
I20250625 07:59:19.762761 10773 raft_consensus.cc:357] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465b6db295aa4ea387b3a1112519672b" member_type: VOTER last_known_addr { host: "127.5.85.69" port: 44751 } } peers { permanent_uuid: "cc5d0afaccda48de8107086e61007220" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 33927 } } peers { permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 } }
I20250625 07:59:19.763839 10773 raft_consensus.cc:383] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:19.764421 10773 raft_consensus.cc:738] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 465b6db295aa4ea387b3a1112519672b, State: Initialized, Role: FOLLOWER
I20250625 07:59:19.764825 10774 raft_consensus.cc:357] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465b6db295aa4ea387b3a1112519672b" member_type: VOTER last_known_addr { host: "127.5.85.69" port: 44751 } } peers { permanent_uuid: "cc5d0afaccda48de8107086e61007220" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 33927 } } peers { permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 } }
I20250625 07:59:19.766073 10774 raft_consensus.cc:383] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:19.766446 10774 raft_consensus.cc:738] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc5d0afaccda48de8107086e61007220, State: Initialized, Role: FOLLOWER
I20250625 07:59:19.765761 10773 consensus_queue.cc:260] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465b6db295aa4ea387b3a1112519672b" member_type: VOTER last_known_addr { host: "127.5.85.69" port: 44751 } } peers { permanent_uuid: "cc5d0afaccda48de8107086e61007220" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 33927 } } peers { permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 } }
I20250625 07:59:19.767510 10774 consensus_queue.cc:260] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465b6db295aa4ea387b3a1112519672b" member_type: VOTER last_known_addr { host: "127.5.85.69" port: 44751 } } peers { permanent_uuid: "cc5d0afaccda48de8107086e61007220" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 33927 } } peers { permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 } }
I20250625 07:59:19.775559 10754 heartbeater.cc:499] Master 127.5.85.126:44819 was elected leader, sending a full tablet report...
I20250625 07:59:19.778784 10774 ts_tablet_manager.cc:1428] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220: Time spent starting tablet: real 0.044s user 0.034s sys 0.008s
I20250625 07:59:19.780516 10773 ts_tablet_manager.cc:1428] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b: Time spent starting tablet: real 0.048s user 0.039s sys 0.004s
I20250625 07:59:19.779839 10775 raft_consensus.cc:357] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465b6db295aa4ea387b3a1112519672b" member_type: VOTER last_known_addr { host: "127.5.85.69" port: 44751 } } peers { permanent_uuid: "cc5d0afaccda48de8107086e61007220" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 33927 } } peers { permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 } }
I20250625 07:59:19.780862 10775 raft_consensus.cc:383] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:19.781277 10775 raft_consensus.cc:738] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b289efe11e547c888b108bee7fbb8b1, State: Initialized, Role: FOLLOWER
I20250625 07:59:19.782269 10775 consensus_queue.cc:260] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465b6db295aa4ea387b3a1112519672b" member_type: VOTER last_known_addr { host: "127.5.85.69" port: 44751 } } peers { permanent_uuid: "cc5d0afaccda48de8107086e61007220" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 33927 } } peers { permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 } }
I20250625 07:59:19.787765 10775 ts_tablet_manager.cc:1428] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1: Time spent starting tablet: real 0.052s user 0.028s sys 0.010s
W20250625 07:59:19.799938 10355 tablet.cc:2378] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 07:59:20.000631 10755 tablet.cc:2378] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 07:59:20.037181 10488 tablet.cc:2378] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:59:20.217617 10779 raft_consensus.cc:491] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:59:20.218128 10779 raft_consensus.cc:513] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465b6db295aa4ea387b3a1112519672b" member_type: VOTER last_known_addr { host: "127.5.85.69" port: 44751 } } peers { permanent_uuid: "cc5d0afaccda48de8107086e61007220" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 33927 } } peers { permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 } }
I20250625 07:59:20.220283 10779 leader_election.cc:290] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cc5d0afaccda48de8107086e61007220 (127.5.85.66:33927), 5b289efe11e547c888b108bee7fbb8b1 (127.5.85.67:39109)
I20250625 07:59:20.231163 10309 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f410c5630e934badab29f22d56a5115f" candidate_uuid: "465b6db295aa4ea387b3a1112519672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5d0afaccda48de8107086e61007220" is_pre_election: true
I20250625 07:59:20.231846 10442 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f410c5630e934badab29f22d56a5115f" candidate_uuid: "465b6db295aa4ea387b3a1112519672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b289efe11e547c888b108bee7fbb8b1" is_pre_election: true
I20250625 07:59:20.232020 10309 raft_consensus.cc:2466] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 465b6db295aa4ea387b3a1112519672b in term 0.
I20250625 07:59:20.232532 10442 raft_consensus.cc:2466] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 465b6db295aa4ea387b3a1112519672b in term 0.
I20250625 07:59:20.233191 10642 leader_election.cc:304] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465b6db295aa4ea387b3a1112519672b, cc5d0afaccda48de8107086e61007220; no voters:
I20250625 07:59:20.233839 10779 raft_consensus.cc:2802] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:59:20.234138 10779 raft_consensus.cc:491] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:59:20.234401 10779 raft_consensus.cc:3058] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:20.238930 10779 raft_consensus.cc:513] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465b6db295aa4ea387b3a1112519672b" member_type: VOTER last_known_addr { host: "127.5.85.69" port: 44751 } } peers { permanent_uuid: "cc5d0afaccda48de8107086e61007220" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 33927 } } peers { permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 } }
I20250625 07:59:20.240549 10779 leader_election.cc:290] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [CANDIDATE]: Term 1 election: Requested vote from peers cc5d0afaccda48de8107086e61007220 (127.5.85.66:33927), 5b289efe11e547c888b108bee7fbb8b1 (127.5.85.67:39109)
I20250625 07:59:20.241300 10309 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f410c5630e934badab29f22d56a5115f" candidate_uuid: "465b6db295aa4ea387b3a1112519672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc5d0afaccda48de8107086e61007220"
I20250625 07:59:20.241436 10442 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f410c5630e934badab29f22d56a5115f" candidate_uuid: "465b6db295aa4ea387b3a1112519672b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b289efe11e547c888b108bee7fbb8b1"
I20250625 07:59:20.241714 10309 raft_consensus.cc:3058] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:20.241855 10442 raft_consensus.cc:3058] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:20.246414 10309 raft_consensus.cc:2466] T f410c5630e934badab29f22d56a5115f P cc5d0afaccda48de8107086e61007220 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 465b6db295aa4ea387b3a1112519672b in term 1.
I20250625 07:59:20.246503 10442 raft_consensus.cc:2466] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 465b6db295aa4ea387b3a1112519672b in term 1.
I20250625 07:59:20.247444 10642 leader_election.cc:304] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 465b6db295aa4ea387b3a1112519672b, cc5d0afaccda48de8107086e61007220; no voters:
I20250625 07:59:20.248039 10779 raft_consensus.cc:2802] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:59:20.249250 10779 raft_consensus.cc:695] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [term 1 LEADER]: Becoming Leader. State: Replica: 465b6db295aa4ea387b3a1112519672b, State: Running, Role: LEADER
I20250625 07:59:20.250121 10779 consensus_queue.cc:237] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465b6db295aa4ea387b3a1112519672b" member_type: VOTER last_known_addr { host: "127.5.85.69" port: 44751 } } peers { permanent_uuid: "cc5d0afaccda48de8107086e61007220" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 33927 } } peers { permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 } }
I20250625 07:59:20.261574 10032 catalog_manager.cc:5582] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b reported cstate change: term changed from 0 to 1, leader changed from <none> to 465b6db295aa4ea387b3a1112519672b (127.5.85.69). New cstate: current_term: 1 leader_uuid: "465b6db295aa4ea387b3a1112519672b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "465b6db295aa4ea387b3a1112519672b" member_type: VOTER last_known_addr { host: "127.5.85.69" port: 44751 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cc5d0afaccda48de8107086e61007220" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 33927 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 } health_report { overall_health: UNKNOWN } } }
I20250625 07:59:20.383395 5461 external_mini_cluster.cc:934] 5 TS(s) registered with all masters
I20250625 07:59:20.387351 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver cc5d0afaccda48de8107086e61007220 to finish bootstrapping
I20250625 07:59:20.399994 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5b289efe11e547c888b108bee7fbb8b1 to finish bootstrapping
I20250625 07:59:20.410826 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 465b6db295aa4ea387b3a1112519672b to finish bootstrapping
I20250625 07:59:20.423650 5461 test_util.cc:276] Using random seed: -762823477
I20250625 07:59:20.448853 5461 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250625 07:59:20.449703 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 10224
I20250625 07:59:20.479940 10442 raft_consensus.cc:1273] T f410c5630e934badab29f22d56a5115f P 5b289efe11e547c888b108bee7fbb8b1 [term 1 FOLLOWER]: Refusing update from remote peer 465b6db295aa4ea387b3a1112519672b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250625 07:59:20.480316 10642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.85.66:33927: connect: Connection refused (error 111)
I20250625 07:59:20.482818 10779 consensus_queue.cc:1035] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b289efe11e547c888b108bee7fbb8b1" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250625 07:59:20.484421 10642 consensus_peers.cc:487] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b -> Peer cc5d0afaccda48de8107086e61007220 (127.5.85.66:33927): Couldn't send request to peer cc5d0afaccda48de8107086e61007220. Status: Network error: Client connection negotiation failed: client connection to 127.5.85.66:33927: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250625 07:59:22.781023 10642 consensus_peers.cc:487] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b -> Peer cc5d0afaccda48de8107086e61007220 (127.5.85.66:33927): Couldn't send request to peer cc5d0afaccda48de8107086e61007220. Status: Network error: Client connection negotiation failed: client connection to 127.5.85.66:33927: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250625 07:59:23.042066 10689 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 07:59:23.048154 10422 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 07:59:23.081648 10155 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 07:59:23.093688 10555 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 07:59:24.868448 10689 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 07:59:24.885892 10155 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 07:59:24.909705 10422 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 07:59:24.945123 10555 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250625 07:59:25.225886 10642 consensus_peers.cc:487] T f410c5630e934badab29f22d56a5115f P 465b6db295aa4ea387b3a1112519672b -> Peer cc5d0afaccda48de8107086e61007220 (127.5.85.66:33927): Couldn't send request to peer cc5d0afaccda48de8107086e61007220. Status: Network error: Client connection negotiation failed: client connection to 127.5.85.66:33927: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250625 07:59:25.648283 10642 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.5.85.66:33927: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250625 07:59:27.328760 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 10091
I20250625 07:59:27.348686 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 10358
I20250625 07:59:27.374791 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 10491
I20250625 07:59:27.394850 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 10624
I20250625 07:59:27.433463 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 9999
2025-06-25T07:59:27Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (18519 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (18519 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250625 07:59:27.490908 5461 test_util.cc:276] Using random seed: -755756224
I20250625 07:59:27.495332 5461 ts_itest-base.cc:115] Starting cluster with:
I20250625 07:59:27.495522 5461 ts_itest-base.cc:116] --------------
I20250625 07:59:27.495643 5461 ts_itest-base.cc:117] 1 tablet servers
I20250625 07:59:27.495750 5461 ts_itest-base.cc:118] 1 replicas per TS
I20250625 07:59:27.495863 5461 ts_itest-base.cc:119] --------------
2025-06-25T07:59:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T07:59:27Z Disabled control of system clock
I20250625 07:59:27.538488 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:33007
--webserver_interface=127.5.85.126
--webserver_port=0
--builtin_ntp_servers=127.5.85.84:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:33007 with env {}
W20250625 07:59:27.842262 10908 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:27.842911 10908 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:27.843343 10908 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:27.874739 10908 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:59:27.875031 10908 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:27.875254 10908 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:59:27.875449 10908 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:59:27.911209 10908 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:33007
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:33007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:27.912520 10908 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:27.914167 10908 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:27.929041 10914 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:27.929031 10917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:27.929301 10915 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:27.930061 10908 server_base.cc:1048] running on GCE node
I20250625 07:59:29.030275 10908 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:29.033317 10908 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:29.034758 10908 hybrid_clock.cc:648] HybridClock initialized: now 1750838369034714 us; error 60 us; skew 500 ppm
I20250625 07:59:29.035583 10908 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:29.046137 10908 webserver.cc:469] Webserver started at http://127.5.85.126:40399/ using document root <none> and password file <none>
I20250625 07:59:29.047166 10908 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:29.047389 10908 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:29.047861 10908 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:29.052294 10908 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "cea9210e1b1b406b9fe9d574bc1563c9"
format_stamp: "Formatted at 2025-06-25 07:59:29 on dist-test-slave-8zlz"
I20250625 07:59:29.053421 10908 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "cea9210e1b1b406b9fe9d574bc1563c9"
format_stamp: "Formatted at 2025-06-25 07:59:29 on dist-test-slave-8zlz"
I20250625 07:59:29.060632 10908 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250625 07:59:29.066411 10924 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:29.067557 10908 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 07:59:29.067883 10908 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
uuid: "cea9210e1b1b406b9fe9d574bc1563c9"
format_stamp: "Formatted at 2025-06-25 07:59:29 on dist-test-slave-8zlz"
I20250625 07:59:29.068215 10908 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:29.120741 10908 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:29.122277 10908 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:29.122786 10908 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:29.190663 10908 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:33007
I20250625 07:59:29.190752 10975 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:33007 every 8 connection(s)
I20250625 07:59:29.193408 10908 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250625 07:59:29.198452 10976 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:29.203541 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 10908
I20250625 07:59:29.203917 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250625 07:59:29.217212 10976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9: Bootstrap starting.
I20250625 07:59:29.223887 10976 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:29.225787 10976 log.cc:826] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:29.230309 10976 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9: No bootstrap required, opened a new log
I20250625 07:59:29.247383 10976 raft_consensus.cc:357] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea9210e1b1b406b9fe9d574bc1563c9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33007 } }
I20250625 07:59:29.248042 10976 raft_consensus.cc:383] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:29.248327 10976 raft_consensus.cc:738] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea9210e1b1b406b9fe9d574bc1563c9, State: Initialized, Role: FOLLOWER
I20250625 07:59:29.249078 10976 consensus_queue.cc:260] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea9210e1b1b406b9fe9d574bc1563c9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33007 } }
I20250625 07:59:29.249739 10976 raft_consensus.cc:397] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:59:29.250065 10976 raft_consensus.cc:491] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:59:29.250396 10976 raft_consensus.cc:3058] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:29.254252 10976 raft_consensus.cc:513] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea9210e1b1b406b9fe9d574bc1563c9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33007 } }
I20250625 07:59:29.254941 10976 leader_election.cc:304] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cea9210e1b1b406b9fe9d574bc1563c9; no voters:
I20250625 07:59:29.256520 10976 leader_election.cc:290] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:59:29.257205 10981 raft_consensus.cc:2802] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:59:29.259230 10981 raft_consensus.cc:695] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [term 1 LEADER]: Becoming Leader. State: Replica: cea9210e1b1b406b9fe9d574bc1563c9, State: Running, Role: LEADER
I20250625 07:59:29.259912 10981 consensus_queue.cc:237] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea9210e1b1b406b9fe9d574bc1563c9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33007 } }
I20250625 07:59:29.260473 10976 sys_catalog.cc:564] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:59:29.269968 10982 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cea9210e1b1b406b9fe9d574bc1563c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea9210e1b1b406b9fe9d574bc1563c9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33007 } } }
I20250625 07:59:29.270675 10982 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [sys.catalog]: This master's current role is: LEADER
I20250625 07:59:29.270882 10983 sys_catalog.cc:455] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cea9210e1b1b406b9fe9d574bc1563c9. Latest consensus state: current_term: 1 leader_uuid: "cea9210e1b1b406b9fe9d574bc1563c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cea9210e1b1b406b9fe9d574bc1563c9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 33007 } } }
I20250625 07:59:29.271522 10983 sys_catalog.cc:458] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9 [sys.catalog]: This master's current role is: LEADER
I20250625 07:59:29.274238 10990 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:59:29.285387 10990 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:59:29.300576 10990 catalog_manager.cc:1349] Generated new cluster ID: 3d3178ff04dd4fee8c79f3ddcec0c5f6
I20250625 07:59:29.300894 10990 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:59:29.315979 10990 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 07:59:29.317605 10990 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:59:29.333782 10990 catalog_manager.cc:5955] T 00000000000000000000000000000000 P cea9210e1b1b406b9fe9d574bc1563c9: Generated new TSK 0
I20250625 07:59:29.334672 10990 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 07:59:29.353183 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:0
--local_ip_for_outbound_sockets=127.5.85.65
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:33007
--builtin_ntp_servers=127.5.85.84:33827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250625 07:59:29.659827 11000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:29.660332 11000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:29.660827 11000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:29.692116 11000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:29.693009 11000 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:59:29.729159 11000 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:33827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:33007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:29.730461 11000 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:29.732378 11000 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:29.748633 11007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:29.749663 11006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:29.748746 11009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:31.164562 11008 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1413 milliseconds
I20250625 07:59:31.164706 11000 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:59:31.166047 11000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:31.168853 11000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:31.170338 11000 hybrid_clock.cc:648] HybridClock initialized: now 1750838371170288 us; error 66 us; skew 500 ppm
I20250625 07:59:31.171280 11000 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:31.177979 11000 webserver.cc:469] Webserver started at http://127.5.85.65:37015/ using document root <none> and password file <none>
I20250625 07:59:31.179026 11000 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:31.179251 11000 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:31.179739 11000 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:31.184377 11000 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "064af2d76b9f45b29a0de1fd84084652"
format_stamp: "Formatted at 2025-06-25 07:59:31 on dist-test-slave-8zlz"
I20250625 07:59:31.185623 11000 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "064af2d76b9f45b29a0de1fd84084652"
format_stamp: "Formatted at 2025-06-25 07:59:31 on dist-test-slave-8zlz"
I20250625 07:59:31.194864 11000 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.001s
I20250625 07:59:31.202531 11016 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:31.203737 11000 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250625 07:59:31.204093 11000 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "064af2d76b9f45b29a0de1fd84084652"
format_stamp: "Formatted at 2025-06-25 07:59:31 on dist-test-slave-8zlz"
I20250625 07:59:31.204452 11000 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:31.254029 11000 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:31.255534 11000 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:31.255950 11000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:31.258404 11000 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:31.262416 11000 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:59:31.262650 11000 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:31.262890 11000 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:59:31.263043 11000 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:31.390395 11000 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:39237
I20250625 07:59:31.390517 11128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:39237 every 8 connection(s)
I20250625 07:59:31.392942 11000 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250625 07:59:31.396718 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 11000
I20250625 07:59:31.397223 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1750838232495789-5461-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250625 07:59:31.418300 11129 heartbeater.cc:344] Connected to a master server at 127.5.85.126:33007
I20250625 07:59:31.418768 11129 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:31.419961 11129 heartbeater.cc:507] Master 127.5.85.126:33007 requested a full tablet report, sending...
I20250625 07:59:31.423146 10940 ts_manager.cc:194] Registered new tserver with Master: 064af2d76b9f45b29a0de1fd84084652 (127.5.85.65:39237)
I20250625 07:59:31.425995 10940 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:33677
I20250625 07:59:31.432402 5461 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250625 07:59:31.464521 10940 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43042:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250625 07:59:31.521435 11064 tablet_service.cc:1468] Processing CreateTablet for tablet 26955c36ee24444c938ba1f0dea8b611 (DEFAULT_TABLE table=TestTable [id=9e161862712d46babb598aa62dd190e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:59:31.522985 11064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26955c36ee24444c938ba1f0dea8b611. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:31.541778 11144 tablet_bootstrap.cc:492] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652: Bootstrap starting.
I20250625 07:59:31.547399 11144 tablet_bootstrap.cc:654] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:31.549114 11144 log.cc:826] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:31.554679 11144 tablet_bootstrap.cc:492] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652: No bootstrap required, opened a new log
I20250625 07:59:31.555145 11144 ts_tablet_manager.cc:1397] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652: Time spent bootstrapping tablet: real 0.014s user 0.013s sys 0.000s
I20250625 07:59:31.576527 11144 raft_consensus.cc:357] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064af2d76b9f45b29a0de1fd84084652" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39237 } }
I20250625 07:59:31.577147 11144 raft_consensus.cc:383] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:31.577376 11144 raft_consensus.cc:738] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 064af2d76b9f45b29a0de1fd84084652, State: Initialized, Role: FOLLOWER
I20250625 07:59:31.578050 11144 consensus_queue.cc:260] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064af2d76b9f45b29a0de1fd84084652" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39237 } }
I20250625 07:59:31.578583 11144 raft_consensus.cc:397] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:59:31.578888 11144 raft_consensus.cc:491] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:59:31.579185 11144 raft_consensus.cc:3058] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:31.583791 11144 raft_consensus.cc:513] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064af2d76b9f45b29a0de1fd84084652" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39237 } }
I20250625 07:59:31.584508 11144 leader_election.cc:304] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 064af2d76b9f45b29a0de1fd84084652; no voters:
I20250625 07:59:31.586134 11144 leader_election.cc:290] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:59:31.586900 11146 raft_consensus.cc:2802] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:59:31.588536 11129 heartbeater.cc:499] Master 127.5.85.126:33007 was elected leader, sending a full tablet report...
I20250625 07:59:31.588999 11146 raft_consensus.cc:695] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [term 1 LEADER]: Becoming Leader. State: Replica: 064af2d76b9f45b29a0de1fd84084652, State: Running, Role: LEADER
I20250625 07:59:31.589397 11144 ts_tablet_manager.cc:1428] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652: Time spent starting tablet: real 0.034s user 0.028s sys 0.004s
I20250625 07:59:31.589896 11146 consensus_queue.cc:237] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064af2d76b9f45b29a0de1fd84084652" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39237 } }
I20250625 07:59:31.602463 10940 catalog_manager.cc:5582] T 26955c36ee24444c938ba1f0dea8b611 P 064af2d76b9f45b29a0de1fd84084652 reported cstate change: term changed from 0 to 1, leader changed from <none> to 064af2d76b9f45b29a0de1fd84084652 (127.5.85.65). New cstate: current_term: 1 leader_uuid: "064af2d76b9f45b29a0de1fd84084652" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "064af2d76b9f45b29a0de1fd84084652" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 39237 } health_report { overall_health: HEALTHY } } }
I20250625 07:59:31.627841 5461 external_mini_cluster.cc:934] 1 TS(s) registered with all masters
I20250625 07:59:31.630722 5461 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 064af2d76b9f45b29a0de1fd84084652 to finish bootstrapping
I20250625 07:59:34.254611 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 11000
I20250625 07:59:34.276199 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 10908
2025-06-25T07:59:34Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (6838 ms)
[----------] 1 test from ListTableCliSimpleParamTest (6838 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250625 07:59:34.329322 5461 test_util.cc:276] Using random seed: -748917809
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (12 ms)
[----------] 1 test from ListTableCliParamTest (12 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-06-25T07:59:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-06-25T07:59:34Z Disabled control of system clock
I20250625 07:59:34.376989 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:37951
--webserver_interface=127.5.85.126
--webserver_port=0
--builtin_ntp_servers=127.5.85.84:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:37951 with env {}
W20250625 07:59:34.669348 11173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:34.669938 11173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:34.670346 11173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:34.702019 11173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:59:34.702308 11173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:34.702509 11173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:59:34.702728 11173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:59:34.738794 11173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:37951
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:37951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:34.740127 11173 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:34.741704 11173 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:34.755676 11182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:34.756107 11179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:34.757295 11173 server_base.cc:1048] running on GCE node
W20250625 07:59:34.756712 11180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:35.848824 11173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:35.851370 11173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:35.852744 11173 hybrid_clock.cc:648] HybridClock initialized: now 1750838375852684 us; error 62 us; skew 500 ppm
I20250625 07:59:35.853555 11173 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:35.865348 11173 webserver.cc:469] Webserver started at http://127.5.85.126:39939/ using document root <none> and password file <none>
I20250625 07:59:35.866282 11173 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:35.866504 11173 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:35.866984 11173 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:35.871523 11173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/instance:
uuid: "fefa5804c499465191b1ec492bce63b9"
format_stamp: "Formatted at 2025-06-25 07:59:35 on dist-test-slave-8zlz"
I20250625 07:59:35.872639 11173 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal/instance:
uuid: "fefa5804c499465191b1ec492bce63b9"
format_stamp: "Formatted at 2025-06-25 07:59:35 on dist-test-slave-8zlz"
I20250625 07:59:35.879915 11173 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20250625 07:59:35.885300 11189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:35.886307 11173 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250625 07:59:35.886658 11173 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
uuid: "fefa5804c499465191b1ec492bce63b9"
format_stamp: "Formatted at 2025-06-25 07:59:35 on dist-test-slave-8zlz"
I20250625 07:59:35.887009 11173 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:35.930348 11173 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:35.931785 11173 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:35.932184 11173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:35.997705 11173 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:37951
I20250625 07:59:35.997771 11240 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:37951 every 8 connection(s)
I20250625 07:59:36.000451 11173 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/info.pb
I20250625 07:59:36.001433 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 11173
I20250625 07:59:36.002003 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal/instance
I20250625 07:59:36.007232 11241 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:36.033105 11241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9: Bootstrap starting.
I20250625 07:59:36.038421 11241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:36.040268 11241 log.cc:826] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:36.044479 11241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9: No bootstrap required, opened a new log
I20250625 07:59:36.061932 11241 raft_consensus.cc:357] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefa5804c499465191b1ec492bce63b9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } }
I20250625 07:59:36.062820 11241 raft_consensus.cc:383] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:36.063077 11241 raft_consensus.cc:738] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fefa5804c499465191b1ec492bce63b9, State: Initialized, Role: FOLLOWER
I20250625 07:59:36.063768 11241 consensus_queue.cc:260] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefa5804c499465191b1ec492bce63b9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } }
I20250625 07:59:36.064298 11241 raft_consensus.cc:397] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:59:36.064581 11241 raft_consensus.cc:491] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:59:36.064891 11241 raft_consensus.cc:3058] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:36.069202 11241 raft_consensus.cc:513] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefa5804c499465191b1ec492bce63b9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } }
I20250625 07:59:36.069965 11241 leader_election.cc:304] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fefa5804c499465191b1ec492bce63b9; no voters:
I20250625 07:59:36.071588 11241 leader_election.cc:290] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:59:36.072315 11246 raft_consensus.cc:2802] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:59:36.074397 11246 raft_consensus.cc:695] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [term 1 LEADER]: Becoming Leader. State: Replica: fefa5804c499465191b1ec492bce63b9, State: Running, Role: LEADER
I20250625 07:59:36.075132 11246 consensus_queue.cc:237] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefa5804c499465191b1ec492bce63b9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } }
I20250625 07:59:36.075634 11241 sys_catalog.cc:564] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:59:36.085947 11247 sys_catalog.cc:455] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fefa5804c499465191b1ec492bce63b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefa5804c499465191b1ec492bce63b9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } } }
I20250625 07:59:36.086772 11247 sys_catalog.cc:458] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [sys.catalog]: This master's current role is: LEADER
I20250625 07:59:36.088117 11248 sys_catalog.cc:455] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fefa5804c499465191b1ec492bce63b9. Latest consensus state: current_term: 1 leader_uuid: "fefa5804c499465191b1ec492bce63b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fefa5804c499465191b1ec492bce63b9" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } } }
I20250625 07:59:36.088922 11248 sys_catalog.cc:458] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9 [sys.catalog]: This master's current role is: LEADER
I20250625 07:59:36.090692 11256 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:59:36.100885 11256 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:59:36.118912 11256 catalog_manager.cc:1349] Generated new cluster ID: 79728844dc1142f3b6a1fe611469d104
I20250625 07:59:36.119310 11256 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:59:36.154753 11256 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 07:59:36.156484 11256 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:59:36.173713 11256 catalog_manager.cc:5955] T 00000000000000000000000000000000 P fefa5804c499465191b1ec492bce63b9: Generated new TSK 0
I20250625 07:59:36.174698 11256 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250625 07:59:36.195227 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:0
--local_ip_for_outbound_sockets=127.5.85.65
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:37951
--builtin_ntp_servers=127.5.85.84:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250625 07:59:36.498538 11265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:36.499083 11265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:36.499612 11265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:36.531323 11265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:36.532186 11265 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:59:36.568245 11265 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=0
--tserver_master_addrs=127.5.85.126:37951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:36.569597 11265 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:36.571384 11265 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:36.589053 11272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:36.590864 11274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:36.589283 11271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:37.702431 11273 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1111 milliseconds
I20250625 07:59:37.702527 11265 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:59:37.703830 11265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:37.706383 11265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:37.707923 11265 hybrid_clock.cc:648] HybridClock initialized: now 1750838377707868 us; error 88 us; skew 500 ppm
I20250625 07:59:37.708739 11265 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:37.715387 11265 webserver.cc:469] Webserver started at http://127.5.85.65:40695/ using document root <none> and password file <none>
I20250625 07:59:37.716341 11265 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:37.716559 11265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:37.717020 11265 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:37.721741 11265 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/instance:
uuid: "037821f13d3e4a039a8729a1256307fe"
format_stamp: "Formatted at 2025-06-25 07:59:37 on dist-test-slave-8zlz"
I20250625 07:59:37.722991 11265 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal/instance:
uuid: "037821f13d3e4a039a8729a1256307fe"
format_stamp: "Formatted at 2025-06-25 07:59:37 on dist-test-slave-8zlz"
I20250625 07:59:37.730549 11265 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250625 07:59:37.736570 11281 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:37.737811 11265 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 07:59:37.738176 11265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal
uuid: "037821f13d3e4a039a8729a1256307fe"
format_stamp: "Formatted at 2025-06-25 07:59:37 on dist-test-slave-8zlz"
I20250625 07:59:37.738519 11265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:37.786751 11265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:37.788326 11265 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:37.788970 11265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:37.792164 11265 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:37.796929 11265 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:59:37.797164 11265 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:37.797397 11265 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:59:37.797549 11265 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:37.954553 11265 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:32813
I20250625 07:59:37.954691 11393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:32813 every 8 connection(s)
I20250625 07:59:37.957350 11265 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/info.pb
I20250625 07:59:37.967116 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 11265
I20250625 07:59:37.967658 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal/instance
I20250625 07:59:37.975021 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.66:0
--local_ip_for_outbound_sockets=127.5.85.66
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:37951
--builtin_ntp_servers=127.5.85.84:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 07:59:37.982901 11394 heartbeater.cc:344] Connected to a master server at 127.5.85.126:37951
I20250625 07:59:37.983434 11394 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:37.984616 11394 heartbeater.cc:507] Master 127.5.85.126:37951 requested a full tablet report, sending...
I20250625 07:59:37.987159 11206 ts_manager.cc:194] Registered new tserver with Master: 037821f13d3e4a039a8729a1256307fe (127.5.85.65:32813)
I20250625 07:59:37.989128 11206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:56769
W20250625 07:59:38.337975 11398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:38.338497 11398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:38.339013 11398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:38.370941 11398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:38.371821 11398 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.66
I20250625 07:59:38.406363 11398 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.85.66
--webserver_port=0
--tserver_master_addrs=127.5.85.126:37951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.66
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:38.407987 11398 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:38.409652 11398 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:38.425194 11405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:38.992740 11394 heartbeater.cc:499] Master 127.5.85.126:37951 was elected leader, sending a full tablet report...
W20250625 07:59:38.425271 11404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:38.425515 11407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:38.427285 11398 server_base.cc:1048] running on GCE node
I20250625 07:59:39.508198 11398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:39.510341 11398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:39.511693 11398 hybrid_clock.cc:648] HybridClock initialized: now 1750838379511656 us; error 45 us; skew 500 ppm
I20250625 07:59:39.512470 11398 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:39.518522 11398 webserver.cc:469] Webserver started at http://127.5.85.66:39019/ using document root <none> and password file <none>
I20250625 07:59:39.519479 11398 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:39.519701 11398 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:39.520159 11398 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:39.525848 11398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/instance:
uuid: "e300568403734babab4e5d9b6581178a"
format_stamp: "Formatted at 2025-06-25 07:59:39 on dist-test-slave-8zlz"
I20250625 07:59:39.526994 11398 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal/instance:
uuid: "e300568403734babab4e5d9b6581178a"
format_stamp: "Formatted at 2025-06-25 07:59:39 on dist-test-slave-8zlz"
I20250625 07:59:39.533995 11398 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250625 07:59:39.539448 11414 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:39.540459 11398 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250625 07:59:39.540767 11398 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal
uuid: "e300568403734babab4e5d9b6581178a"
format_stamp: "Formatted at 2025-06-25 07:59:39 on dist-test-slave-8zlz"
I20250625 07:59:39.541106 11398 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:39.597841 11398 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:39.599398 11398 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:39.599862 11398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:39.602347 11398 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:39.606477 11398 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:59:39.606724 11398 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250625 07:59:39.606978 11398 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:59:39.607161 11398 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:39.735425 11398 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.66:39545
I20250625 07:59:39.735532 11526 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.66:39545 every 8 connection(s)
I20250625 07:59:39.737905 11398 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/info.pb
I20250625 07:59:39.744915 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 11398
I20250625 07:59:39.745436 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal/instance
I20250625 07:59:39.753034 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.67:0
--local_ip_for_outbound_sockets=127.5.85.67
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:37951
--builtin_ntp_servers=127.5.85.84:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 07:59:39.761423 11527 heartbeater.cc:344] Connected to a master server at 127.5.85.126:37951
I20250625 07:59:39.761894 11527 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:39.763036 11527 heartbeater.cc:507] Master 127.5.85.126:37951 requested a full tablet report, sending...
I20250625 07:59:39.765280 11206 ts_manager.cc:194] Registered new tserver with Master: e300568403734babab4e5d9b6581178a (127.5.85.66:39545)
I20250625 07:59:39.766580 11206 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.66:59373
W20250625 07:59:40.066202 11531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:40.066753 11531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:40.067271 11531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:40.098690 11531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:40.099552 11531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.67
I20250625 07:59:40.133905 11531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.85.67
--webserver_port=0
--tserver_master_addrs=127.5.85.126:37951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.67
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:40.135249 11531 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:40.136893 11531 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:40.152191 11537 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:40.769886 11527 heartbeater.cc:499] Master 127.5.85.126:37951 was elected leader, sending a full tablet report...
W20250625 07:59:40.152195 11540 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:40.152720 11538 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:41.222105 11539 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250625 07:59:41.222226 11531 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:59:41.225908 11531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:41.228210 11531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:41.229591 11531 hybrid_clock.cc:648] HybridClock initialized: now 1750838381229550 us; error 53 us; skew 500 ppm
I20250625 07:59:41.230419 11531 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:41.241483 11531 webserver.cc:469] Webserver started at http://127.5.85.67:35285/ using document root <none> and password file <none>
I20250625 07:59:41.242451 11531 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:41.242703 11531 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:41.243176 11531 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:41.247673 11531 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/instance:
uuid: "f39d2c8665264ac1a266072963a58329"
format_stamp: "Formatted at 2025-06-25 07:59:41 on dist-test-slave-8zlz"
I20250625 07:59:41.248814 11531 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal/instance:
uuid: "f39d2c8665264ac1a266072963a58329"
format_stamp: "Formatted at 2025-06-25 07:59:41 on dist-test-slave-8zlz"
I20250625 07:59:41.255861 11531 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250625 07:59:41.261441 11547 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:41.262533 11531 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250625 07:59:41.262900 11531 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal
uuid: "f39d2c8665264ac1a266072963a58329"
format_stamp: "Formatted at 2025-06-25 07:59:41 on dist-test-slave-8zlz"
I20250625 07:59:41.263250 11531 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:41.314697 11531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:41.316169 11531 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:41.316629 11531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:41.319271 11531 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:41.323442 11531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250625 07:59:41.323665 11531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:41.323930 11531 ts_tablet_manager.cc:610] Registered 0 tablets
I20250625 07:59:41.324088 11531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:41.452526 11531 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.67:39509
I20250625 07:59:41.452718 11659 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.67:39509 every 8 connection(s)
I20250625 07:59:41.455091 11531 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/info.pb
I20250625 07:59:41.456391 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 11531
I20250625 07:59:41.456930 5461 external_mini_cluster.cc:1427] Reading /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal/instance
I20250625 07:59:41.476303 11660 heartbeater.cc:344] Connected to a master server at 127.5.85.126:37951
I20250625 07:59:41.476826 11660 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:41.478133 11660 heartbeater.cc:507] Master 127.5.85.126:37951 requested a full tablet report, sending...
I20250625 07:59:41.480321 11205 ts_manager.cc:194] Registered new tserver with Master: f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509)
I20250625 07:59:41.481652 11205 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.67:32867
I20250625 07:59:41.493340 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:59:41.519917 5461 test_util.cc:276] Using random seed: -741727216
I20250625 07:59:41.568717 11205 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60538:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250625 07:59:41.571377 11205 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 07:59:41.631805 11595 tablet_service.cc:1468] Processing CreateTablet for tablet 7acda0f994e045acb1a45aafb8c890f1 (DEFAULT_TABLE table=pre_rebuild [id=89117d7273244f78964ecf6de4cebece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:59:41.631822 11462 tablet_service.cc:1468] Processing CreateTablet for tablet 7acda0f994e045acb1a45aafb8c890f1 (DEFAULT_TABLE table=pre_rebuild [id=89117d7273244f78964ecf6de4cebece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:59:41.633955 11595 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7acda0f994e045acb1a45aafb8c890f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:41.634853 11462 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7acda0f994e045acb1a45aafb8c890f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:41.634742 11329 tablet_service.cc:1468] Processing CreateTablet for tablet 7acda0f994e045acb1a45aafb8c890f1 (DEFAULT_TABLE table=pre_rebuild [id=89117d7273244f78964ecf6de4cebece]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 07:59:41.636226 11329 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7acda0f994e045acb1a45aafb8c890f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:41.659857 11684 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Bootstrap starting.
I20250625 07:59:41.662693 11685 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Bootstrap starting.
I20250625 07:59:41.665128 11686 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Bootstrap starting.
I20250625 07:59:41.670126 11685 tablet_bootstrap.cc:654] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:41.670126 11684 tablet_bootstrap.cc:654] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:41.670526 11686 tablet_bootstrap.cc:654] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:41.672348 11686 log.cc:826] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:41.672485 11684 log.cc:826] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:41.672482 11685 log.cc:826] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:41.677460 11686 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: No bootstrap required, opened a new log
I20250625 07:59:41.677908 11686 ts_tablet_manager.cc:1397] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.003s
I20250625 07:59:41.678118 11684 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: No bootstrap required, opened a new log
I20250625 07:59:41.678670 11684 ts_tablet_manager.cc:1397] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.011s
I20250625 07:59:41.680142 11685 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: No bootstrap required, opened a new log
I20250625 07:59:41.680621 11685 ts_tablet_manager.cc:1397] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.008s
I20250625 07:59:41.695439 11686 raft_consensus.cc:357] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:41.696353 11686 raft_consensus.cc:383] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:41.696642 11686 raft_consensus.cc:738] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39d2c8665264ac1a266072963a58329, State: Initialized, Role: FOLLOWER
I20250625 07:59:41.697650 11686 consensus_queue.cc:260] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:41.705523 11684 raft_consensus.cc:357] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:41.706502 11684 raft_consensus.cc:383] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:41.706823 11684 raft_consensus.cc:738] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037821f13d3e4a039a8729a1256307fe, State: Initialized, Role: FOLLOWER
I20250625 07:59:41.707743 11684 consensus_queue.cc:260] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:41.708839 11686 ts_tablet_manager.cc:1428] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Time spent starting tablet: real 0.031s user 0.028s sys 0.001s
I20250625 07:59:41.709666 11660 heartbeater.cc:499] Master 127.5.85.126:37951 was elected leader, sending a full tablet report...
I20250625 07:59:41.710099 11685 raft_consensus.cc:357] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:41.711083 11685 raft_consensus.cc:383] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:41.711375 11685 raft_consensus.cc:738] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e300568403734babab4e5d9b6581178a, State: Initialized, Role: FOLLOWER
W20250625 07:59:41.711992 11661 tablet.cc:2378] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:59:41.712405 11685 consensus_queue.cc:260] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:41.715562 11684 ts_tablet_manager.cc:1428] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Time spent starting tablet: real 0.037s user 0.026s sys 0.009s
W20250625 07:59:41.716477 11395 tablet.cc:2378] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:59:41.718231 11685 ts_tablet_manager.cc:1428] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Time spent starting tablet: real 0.037s user 0.022s sys 0.013s
W20250625 07:59:41.744295 11528 tablet.cc:2378] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 07:59:41.863997 11690 raft_consensus.cc:491] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:59:41.864523 11690 raft_consensus.cc:513] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:41.866920 11690 leader_election.cc:290] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 037821f13d3e4a039a8729a1256307fe (127.5.85.65:32813), e300568403734babab4e5d9b6581178a (127.5.85.66:39545)
I20250625 07:59:41.875949 11482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7acda0f994e045acb1a45aafb8c890f1" candidate_uuid: "f39d2c8665264ac1a266072963a58329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e300568403734babab4e5d9b6581178a" is_pre_election: true
I20250625 07:59:41.876724 11482 raft_consensus.cc:2466] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f39d2c8665264ac1a266072963a58329 in term 0.
I20250625 07:59:41.878039 11550 leader_election.cc:304] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e300568403734babab4e5d9b6581178a, f39d2c8665264ac1a266072963a58329; no voters:
I20250625 07:59:41.878435 11349 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7acda0f994e045acb1a45aafb8c890f1" candidate_uuid: "f39d2c8665264ac1a266072963a58329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037821f13d3e4a039a8729a1256307fe" is_pre_election: true
I20250625 07:59:41.878942 11690 raft_consensus.cc:2802] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 07:59:41.879248 11690 raft_consensus.cc:491] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 07:59:41.879231 11349 raft_consensus.cc:2466] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f39d2c8665264ac1a266072963a58329 in term 0.
I20250625 07:59:41.879498 11690 raft_consensus.cc:3058] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:41.883956 11690 raft_consensus.cc:513] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:41.885793 11690 leader_election.cc:290] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [CANDIDATE]: Term 1 election: Requested vote from peers 037821f13d3e4a039a8729a1256307fe (127.5.85.65:32813), e300568403734babab4e5d9b6581178a (127.5.85.66:39545)
I20250625 07:59:41.886801 11349 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7acda0f994e045acb1a45aafb8c890f1" candidate_uuid: "f39d2c8665264ac1a266072963a58329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "037821f13d3e4a039a8729a1256307fe"
I20250625 07:59:41.887346 11349 raft_consensus.cc:3058] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:41.888248 11482 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7acda0f994e045acb1a45aafb8c890f1" candidate_uuid: "f39d2c8665264ac1a266072963a58329" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e300568403734babab4e5d9b6581178a"
I20250625 07:59:41.888772 11482 raft_consensus.cc:3058] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:41.893997 11349 raft_consensus.cc:2466] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f39d2c8665264ac1a266072963a58329 in term 1.
I20250625 07:59:41.895226 11482 raft_consensus.cc:2466] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f39d2c8665264ac1a266072963a58329 in term 1.
I20250625 07:59:41.895349 11549 leader_election.cc:304] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037821f13d3e4a039a8729a1256307fe, f39d2c8665264ac1a266072963a58329; no voters:
I20250625 07:59:41.896144 11690 raft_consensus.cc:2802] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:59:41.897804 11690 raft_consensus.cc:695] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 1 LEADER]: Becoming Leader. State: Replica: f39d2c8665264ac1a266072963a58329, State: Running, Role: LEADER
I20250625 07:59:41.898829 11690 consensus_queue.cc:237] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:41.910990 11206 catalog_manager.cc:5582] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 reported cstate change: term changed from 0 to 1, leader changed from <none> to f39d2c8665264ac1a266072963a58329 (127.5.85.67). New cstate: current_term: 1 leader_uuid: "f39d2c8665264ac1a266072963a58329" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } health_report { overall_health: UNKNOWN } } }
I20250625 07:59:42.117892 11482 raft_consensus.cc:1273] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Refusing update from remote peer f39d2c8665264ac1a266072963a58329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:59:42.118131 11349 raft_consensus.cc:1273] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Refusing update from remote peer f39d2c8665264ac1a266072963a58329: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 07:59:42.119585 11694 consensus_queue.cc:1035] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 07:59:42.120316 11690 consensus_queue.cc:1035] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [LEADER]: Connected to new peer: Peer: permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 07:59:42.141690 11702 mvcc.cc:204] Tried to move back new op lower bound from 7171434013138599936 to 7171434012266582016. Current Snapshot: MvccSnapshot[applied={T|T < 7171434013138599936}]
I20250625 07:59:42.148746 11704 mvcc.cc:204] Tried to move back new op lower bound from 7171434013138599936 to 7171434012266582016. Current Snapshot: MvccSnapshot[applied={T|T < 7171434013138599936}]
I20250625 07:59:42.192575 11703 mvcc.cc:204] Tried to move back new op lower bound from 7171434013138599936 to 7171434012266582016. Current Snapshot: MvccSnapshot[applied={T|T < 7171434013138599936}]
I20250625 07:59:47.259267 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 11173
W20250625 07:59:47.280812 11551 connection.cc:537] client connection to 127.5.85.126:37951 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 07:59:47.281888 11660 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:37951 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250625 07:59:47.299974 11527 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:37951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:37951: connect: Connection refused (error 111)
W20250625 07:59:47.640306 11735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:47.640916 11735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:47.672253 11735 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250625 07:59:48.221140 11394 heartbeater.cc:646] Failed to heartbeat to 127.5.85.126:37951 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.5.85.126:37951: connect: Connection refused (error 111)
W20250625 07:59:49.023907 11735 thread.cc:640] rpc reactor (reactor) Time spent creating pthread: real 1.310s user 0.595s sys 0.705s
W20250625 07:59:49.024214 11735 thread.cc:606] rpc reactor (reactor) Time spent starting thread: real 1.310s user 0.595s sys 0.705s
I20250625 07:59:49.158280 11735 minidump.cc:252] Setting minidump size limit to 20M
I20250625 07:59:49.160776 11735 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:49.161968 11735 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:49.173492 11768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:49.173588 11769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:49.275002 11735 server_base.cc:1048] running on GCE node
W20250625 07:59:49.275048 11772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:49.276384 11735 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250625 07:59:49.276940 11735 hybrid_clock.cc:648] HybridClock initialized: now 1750838389276915 us; error 317016 us; skew 500 ppm
I20250625 07:59:49.277609 11735 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:49.281968 11735 webserver.cc:469] Webserver started at http://0.0.0.0:36951/ using document root <none> and password file <none>
I20250625 07:59:49.282811 11735 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:49.283027 11735 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:49.283413 11735 server_base.cc:896] This appears to be a new deployment of Kudu; creating new FS layout
I20250625 07:59:49.287643 11735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/instance:
uuid: "b024f4bf98434878977ea6818d7f9063"
format_stamp: "Formatted at 2025-06-25 07:59:49 on dist-test-slave-8zlz"
I20250625 07:59:49.288759 11735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal/instance:
uuid: "b024f4bf98434878977ea6818d7f9063"
format_stamp: "Formatted at 2025-06-25 07:59:49 on dist-test-slave-8zlz"
I20250625 07:59:49.294672 11735 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20250625 07:59:49.299922 11777 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:49.300815 11735 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250625 07:59:49.301149 11735 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
uuid: "b024f4bf98434878977ea6818d7f9063"
format_stamp: "Formatted at 2025-06-25 07:59:49 on dist-test-slave-8zlz"
I20250625 07:59:49.301496 11735 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:49.435271 11735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:49.436725 11735 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:49.437206 11735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:49.442219 11735 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 07:59:49.457950 11735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: Bootstrap starting.
I20250625 07:59:49.464912 11735 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: Neither blocks nor log segments found. Creating new log.
I20250625 07:59:49.467216 11735 log.cc:826] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:49.472488 11735 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: No bootstrap required, opened a new log
I20250625 07:59:49.496904 11735 raft_consensus.cc:357] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER }
I20250625 07:59:49.497586 11735 raft_consensus.cc:383] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 07:59:49.497870 11735 raft_consensus.cc:738] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b024f4bf98434878977ea6818d7f9063, State: Initialized, Role: FOLLOWER
I20250625 07:59:49.498549 11735 consensus_queue.cc:260] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER }
I20250625 07:59:49.499089 11735 raft_consensus.cc:397] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:59:49.499310 11735 raft_consensus.cc:491] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:59:49.499557 11735 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 0 FOLLOWER]: Advancing to term 1
I20250625 07:59:49.503803 11735 raft_consensus.cc:513] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER }
I20250625 07:59:49.504590 11735 leader_election.cc:304] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b024f4bf98434878977ea6818d7f9063; no voters:
I20250625 07:59:49.506234 11735 leader_election.cc:290] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250625 07:59:49.506505 11786 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 1 FOLLOWER]: Leader election won for term 1
I20250625 07:59:49.509132 11786 raft_consensus.cc:695] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 1 LEADER]: Becoming Leader. State: Replica: b024f4bf98434878977ea6818d7f9063, State: Running, Role: LEADER
I20250625 07:59:49.509995 11786 consensus_queue.cc:237] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER }
I20250625 07:59:49.518030 11788 sys_catalog.cc:455] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b024f4bf98434878977ea6818d7f9063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER } }
I20250625 07:59:49.518788 11788 sys_catalog.cc:458] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [sys.catalog]: This master's current role is: LEADER
I20250625 07:59:49.519096 11787 sys_catalog.cc:455] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b024f4bf98434878977ea6818d7f9063. Latest consensus state: current_term: 1 leader_uuid: "b024f4bf98434878977ea6818d7f9063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER } }
I20250625 07:59:49.519536 11787 sys_catalog.cc:458] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [sys.catalog]: This master's current role is: LEADER
I20250625 07:59:49.532708 11735 tablet_replica.cc:331] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: stopping tablet replica
I20250625 07:59:49.533296 11735 raft_consensus.cc:2241] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 1 LEADER]: Raft consensus shutting down.
I20250625 07:59:49.533742 11735 raft_consensus.cc:2270] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250625 07:59:49.536072 11735 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250625 07:59:49.536535 11735 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250625 07:59:49.580366 11735 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250625 07:59:50.609180 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 11265
I20250625 07:59:50.638398 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 11398
W20250625 07:59:50.667212 11550 connection.cc:537] client connection to 127.5.85.66:39545 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250625 07:59:50.667464 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 11531
I20250625 07:59:50.702956 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:37951
--webserver_interface=127.5.85.126
--webserver_port=39939
--builtin_ntp_servers=127.5.85.84:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.5.85.126:37951 with env {}
W20250625 07:59:51.002718 11796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:51.003283 11796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:51.003696 11796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:51.035024 11796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250625 07:59:51.035315 11796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:51.035521 11796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250625 07:59:51.035718 11796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250625 07:59:51.070959 11796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.5.85.126:37951
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.5.85.126:37951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.5.85.126
--webserver_port=39939
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:51.072244 11796 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:51.073884 11796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:51.088212 11803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:51.088346 11802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:51.090188 11796 server_base.cc:1048] running on GCE node
W20250625 07:59:51.088338 11805 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:52.185699 11796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:52.188705 11796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:52.190171 11796 hybrid_clock.cc:648] HybridClock initialized: now 1750838392190140 us; error 63 us; skew 500 ppm
I20250625 07:59:52.191064 11796 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:52.197613 11796 webserver.cc:469] Webserver started at http://127.5.85.126:39939/ using document root <none> and password file <none>
I20250625 07:59:52.198558 11796 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:52.198839 11796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:52.206566 11796 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250625 07:59:52.211056 11812 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:52.212065 11796 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250625 07:59:52.212407 11796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
uuid: "b024f4bf98434878977ea6818d7f9063"
format_stamp: "Formatted at 2025-06-25 07:59:49 on dist-test-slave-8zlz"
I20250625 07:59:52.214282 11796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:52.267292 11796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:52.268767 11796 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:52.269248 11796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:52.337580 11796 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.126:37951
I20250625 07:59:52.337687 11863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.126:37951 every 8 connection(s)
I20250625 07:59:52.340339 11796 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/info.pb
I20250625 07:59:52.348395 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 11796
I20250625 07:59:52.350390 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.65:32813
--local_ip_for_outbound_sockets=127.5.85.65
--tserver_master_addrs=127.5.85.126:37951
--webserver_port=40695
--webserver_interface=127.5.85.65
--builtin_ntp_servers=127.5.85.84:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 07:59:52.350942 11864 sys_catalog.cc:263] Verifying existing consensus state
I20250625 07:59:52.365545 11864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: Bootstrap starting.
I20250625 07:59:52.375259 11864 log.cc:826] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:52.387387 11864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:59:52.388200 11864 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: Bootstrap complete.
I20250625 07:59:52.408910 11864 raft_consensus.cc:357] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } }
I20250625 07:59:52.409631 11864 raft_consensus.cc:738] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b024f4bf98434878977ea6818d7f9063, State: Initialized, Role: FOLLOWER
I20250625 07:59:52.410367 11864 consensus_queue.cc:260] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } }
I20250625 07:59:52.410909 11864 raft_consensus.cc:397] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250625 07:59:52.411180 11864 raft_consensus.cc:491] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250625 07:59:52.411489 11864 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 1 FOLLOWER]: Advancing to term 2
I20250625 07:59:52.415772 11864 raft_consensus.cc:513] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } }
I20250625 07:59:52.416455 11864 leader_election.cc:304] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b024f4bf98434878977ea6818d7f9063; no voters:
I20250625 07:59:52.418778 11864 leader_election.cc:290] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250625 07:59:52.419107 11868 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 2 FOLLOWER]: Leader election won for term 2
I20250625 07:59:52.423161 11868 raft_consensus.cc:695] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [term 2 LEADER]: Becoming Leader. State: Replica: b024f4bf98434878977ea6818d7f9063, State: Running, Role: LEADER
I20250625 07:59:52.423882 11864 sys_catalog.cc:564] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [sys.catalog]: configured and running, proceeding with master startup.
I20250625 07:59:52.424144 11868 consensus_queue.cc:237] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } }
I20250625 07:59:52.436419 11869 sys_catalog.cc:455] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b024f4bf98434878977ea6818d7f9063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } } }
I20250625 07:59:52.437140 11869 sys_catalog.cc:458] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [sys.catalog]: This master's current role is: LEADER
I20250625 07:59:52.439106 11870 sys_catalog.cc:455] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b024f4bf98434878977ea6818d7f9063. Latest consensus state: current_term: 2 leader_uuid: "b024f4bf98434878977ea6818d7f9063" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b024f4bf98434878977ea6818d7f9063" member_type: VOTER last_known_addr { host: "127.5.85.126" port: 37951 } } }
I20250625 07:59:52.439908 11870 sys_catalog.cc:458] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063 [sys.catalog]: This master's current role is: LEADER
I20250625 07:59:52.451059 11876 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250625 07:59:52.464567 11876 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=fdb6782a411740f089736018542bf1c9]
I20250625 07:59:52.472071 11876 tablet_loader.cc:96] loaded metadata for tablet 7acda0f994e045acb1a45aafb8c890f1 (table pre_rebuild [id=fdb6782a411740f089736018542bf1c9])
I20250625 07:59:52.473726 11876 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250625 07:59:52.494443 11885 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250625 07:59:52.497855 11876 catalog_manager.cc:1349] Generated new cluster ID: 1be510a325844dd1b171aa84dfe66804
I20250625 07:59:52.498123 11876 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250625 07:59:52.540107 11876 catalog_manager.cc:1372] Generated new certificate authority record
I20250625 07:59:52.541553 11876 catalog_manager.cc:1506] Loading token signing keys...
I20250625 07:59:52.556423 11876 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: Generated new TSK 0
I20250625 07:59:52.557341 11876 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250625 07:59:52.706974 11866 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:52.707469 11866 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:52.707950 11866 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:52.740041 11866 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:52.740881 11866 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.65
I20250625 07:59:52.776118 11866 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.65:32813
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.5.85.65
--webserver_port=40695
--tserver_master_addrs=127.5.85.126:37951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.65
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:52.777410 11866 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:52.779151 11866 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:52.796241 11893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:52.796298 11895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:52.796447 11892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:54.208102 11894 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1409 milliseconds
I20250625 07:59:54.208227 11866 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250625 07:59:54.209473 11866 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:54.212139 11866 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:54.213604 11866 hybrid_clock.cc:648] HybridClock initialized: now 1750838394213535 us; error 87 us; skew 500 ppm
I20250625 07:59:54.214450 11866 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:54.228271 11866 webserver.cc:469] Webserver started at http://127.5.85.65:40695/ using document root <none> and password file <none>
I20250625 07:59:54.229671 11866 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:54.229987 11866 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:54.241161 11866 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20250625 07:59:54.247408 11903 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:54.248616 11866 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250625 07:59:54.248962 11866 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal
uuid: "037821f13d3e4a039a8729a1256307fe"
format_stamp: "Formatted at 2025-06-25 07:59:37 on dist-test-slave-8zlz"
I20250625 07:59:54.251049 11866 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:54.313195 11866 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:54.314821 11866 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:54.315297 11866 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:54.317930 11866 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:54.324744 11910 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 07:59:54.333009 11866 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 07:59:54.333271 11866 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250625 07:59:54.333539 11866 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 07:59:54.338577 11866 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 07:59:54.338939 11866 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.006s sys 0.000s
I20250625 07:59:54.339335 11910 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Bootstrap starting.
I20250625 07:59:54.512753 11866 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.65:32813
I20250625 07:59:54.512876 12016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.65:32813 every 8 connection(s)
I20250625 07:59:54.516479 11866 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/info.pb
I20250625 07:59:54.525000 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 11866
I20250625 07:59:54.526995 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.66:39545
--local_ip_for_outbound_sockets=127.5.85.66
--tserver_master_addrs=127.5.85.126:37951
--webserver_port=39019
--webserver_interface=127.5.85.66
--builtin_ntp_servers=127.5.85.84:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 07:59:54.550158 12017 heartbeater.cc:344] Connected to a master server at 127.5.85.126:37951
I20250625 07:59:54.550745 12017 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:54.552131 12017 heartbeater.cc:507] Master 127.5.85.126:37951 requested a full tablet report, sending...
I20250625 07:59:54.557287 11829 ts_manager.cc:194] Registered new tserver with Master: 037821f13d3e4a039a8729a1256307fe (127.5.85.65:32813)
I20250625 07:59:54.564715 11829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.65:48759
I20250625 07:59:54.615999 11910 log.cc:826] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Log is configured to *not* fsync() on all Append() calls
W20250625 07:59:54.839488 12021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:54.840020 12021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:54.840538 12021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250625 07:59:54.872771 12021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:54.873642 12021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.66
I20250625 07:59:54.910876 12021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.66:39545
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.5.85.66
--webserver_port=39019
--tserver_master_addrs=127.5.85.126:37951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.66
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:54.912174 12021 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:54.913770 12021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:54.930877 12028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:55.569290 12017 heartbeater.cc:499] Master 127.5.85.126:37951 was elected leader, sending a full tablet report...
W20250625 07:59:54.931795 12031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250625 07:59:56.021675 12030 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1092 milliseconds
I20250625 07:59:56.021782 12021 server_base.cc:1043] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250625 07:59:54.930965 12029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:56.025974 12021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:56.028613 12021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:56.030088 12021 hybrid_clock.cc:648] HybridClock initialized: now 1750838396030024 us; error 79 us; skew 500 ppm
I20250625 07:59:56.030972 12021 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:56.037696 12021 webserver.cc:469] Webserver started at http://127.5.85.66:39019/ using document root <none> and password file <none>
I20250625 07:59:56.038596 12021 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:56.038864 12021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:56.046764 12021 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20250625 07:59:56.051319 12038 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:56.052373 12021 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250625 07:59:56.052670 12021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal
uuid: "e300568403734babab4e5d9b6581178a"
format_stamp: "Formatted at 2025-06-25 07:59:39 on dist-test-slave-8zlz"
I20250625 07:59:56.054666 12021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:56.127498 12021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:56.128978 12021 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:56.129428 12021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:56.132382 12021 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:56.138377 12045 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 07:59:56.145948 12021 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 07:59:56.146181 12021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250625 07:59:56.146456 12021 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 07:59:56.151331 12021 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 07:59:56.151607 12021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250625 07:59:56.152096 12045 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Bootstrap starting.
I20250625 07:59:56.376803 12021 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.66:39545
I20250625 07:59:56.376958 12151 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.66:39545 every 8 connection(s)
I20250625 07:59:56.379853 12021 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/info.pb
I20250625 07:59:56.389899 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 12021
I20250625 07:59:56.392131 5461 external_mini_cluster.cc:1351] Running /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
/tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.5.85.67:39509
--local_ip_for_outbound_sockets=127.5.85.67
--tserver_master_addrs=127.5.85.126:37951
--webserver_port=35285
--webserver_interface=127.5.85.67
--builtin_ntp_servers=127.5.85.84:41631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250625 07:59:56.416982 12152 heartbeater.cc:344] Connected to a master server at 127.5.85.126:37951
I20250625 07:59:56.417476 12152 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:56.418686 12152 heartbeater.cc:507] Master 127.5.85.126:37951 requested a full tablet report, sending...
I20250625 07:59:56.422508 11829 ts_manager.cc:194] Registered new tserver with Master: e300568403734babab4e5d9b6581178a (127.5.85.66:39545)
I20250625 07:59:56.425554 11829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.66:51253
I20250625 07:59:56.463687 12045 log.cc:826] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Log is configured to *not* fsync() on all Append() calls
W20250625 07:59:56.725644 12156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250625 07:59:56.726143 12156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250625 07:59:56.726676 12156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250625 07:59:56.758333 11910 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:59:56.759369 11910 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Bootstrap complete.
W20250625 07:59:56.759433 12156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250625 07:59:56.760241 12156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.5.85.67
I20250625 07:59:56.761020 11910 ts_tablet_manager.cc:1397] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Time spent bootstrapping tablet: real 2.422s user 2.340s sys 0.064s
I20250625 07:59:56.777837 11910 raft_consensus.cc:357] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:56.780872 11910 raft_consensus.cc:738] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037821f13d3e4a039a8729a1256307fe, State: Initialized, Role: FOLLOWER
I20250625 07:59:56.781719 11910 consensus_queue.cc:260] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:56.785332 11910 ts_tablet_manager.cc:1428] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Time spent starting tablet: real 0.024s user 0.025s sys 0.000s
I20250625 07:59:56.798826 12156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.5.85.84:41631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.5.85.67:39509
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.5.85.67
--webserver_port=35285
--tserver_master_addrs=127.5.85.126:37951
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.5.85.67
--log_dir=/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 6d72d4a242076468501f3430b9a2cd050c634be2
build type FASTDEBUG
built by None at 25 Jun 2025 07:43:21 UTC on 5fd53c4cbb9d
build id 6761
TSAN enabled
I20250625 07:59:56.800076 12156 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250625 07:59:56.801743 12156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250625 07:59:56.819092 12164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:57.430052 12152 heartbeater.cc:499] Master 127.5.85.126:37951 was elected leader, sending a full tablet report...
W20250625 07:59:56.819101 12165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:56.821501 12156 server_base.cc:1048] running on GCE node
W20250625 07:59:56.821409 12167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250625 07:59:57.912709 12156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250625 07:59:57.916392 12156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250625 07:59:57.917937 12156 hybrid_clock.cc:648] HybridClock initialized: now 1750838397917890 us; error 51 us; skew 500 ppm
I20250625 07:59:57.918893 12156 server_base.cc:848] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250625 07:59:57.926556 12156 webserver.cc:469] Webserver started at http://127.5.85.67:35285/ using document root <none> and password file <none>
I20250625 07:59:57.927528 12156 fs_manager.cc:362] Metadata directory not provided
I20250625 07:59:57.927737 12156 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250625 07:59:57.935794 12156 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.008s sys 0.000s
I20250625 07:59:57.940429 12174 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250625 07:59:57.941497 12156 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250625 07:59:57.941787 12156 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data,/tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal
uuid: "f39d2c8665264ac1a266072963a58329"
format_stamp: "Formatted at 2025-06-25 07:59:41 on dist-test-slave-8zlz"
I20250625 07:59:57.943734 12156 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250625 07:59:57.992707 12156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250625 07:59:57.994122 12156 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250625 07:59:57.994530 12156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250625 07:59:57.997305 12156 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250625 07:59:58.003471 12181 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250625 07:59:58.010836 12156 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250625 07:59:58.011051 12156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250625 07:59:58.011293 12156 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250625 07:59:58.015851 12156 ts_tablet_manager.cc:610] Registered 1 tablets
I20250625 07:59:58.016103 12156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.001s
I20250625 07:59:58.016484 12181 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Bootstrap starting.
I20250625 07:59:58.186345 12156 rpc_server.cc:307] RPC server started. Bound to: 127.5.85.67:39509
I20250625 07:59:58.186717 12287 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.5.85.67:39509 every 8 connection(s)
I20250625 07:59:58.190140 12156 server_base.cc:1180] Dumped server information to /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/info.pb
I20250625 07:59:58.197667 5461 external_mini_cluster.cc:1413] Started /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu as pid 12156
I20250625 07:59:58.240960 12288 heartbeater.cc:344] Connected to a master server at 127.5.85.126:37951
I20250625 07:59:58.241506 12288 heartbeater.cc:461] Registering TS with master...
I20250625 07:59:58.242867 12288 heartbeater.cc:507] Master 127.5.85.126:37951 requested a full tablet report, sending...
I20250625 07:59:58.246718 11829 ts_manager.cc:194] Registered new tserver with Master: f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509)
I20250625 07:59:58.249668 11829 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.5.85.67:42133
I20250625 07:59:58.259274 5461 external_mini_cluster.cc:934] 3 TS(s) registered with all masters
I20250625 07:59:58.330364 12181 log.cc:826] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Log is configured to *not* fsync() on all Append() calls
I20250625 07:59:58.451606 12300 raft_consensus.cc:491] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 07:59:58.452210 12300 raft_consensus.cc:513] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:58.455472 12300 leader_election.cc:290] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509), e300568403734babab4e5d9b6581178a (127.5.85.66:39545)
I20250625 07:59:58.487898 12107 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7acda0f994e045acb1a45aafb8c890f1" candidate_uuid: "037821f13d3e4a039a8729a1256307fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "e300568403734babab4e5d9b6581178a" is_pre_election: true
I20250625 07:59:58.487902 12243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7acda0f994e045acb1a45aafb8c890f1" candidate_uuid: "037821f13d3e4a039a8729a1256307fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f39d2c8665264ac1a266072963a58329" is_pre_election: true
W20250625 07:59:58.501045 11906 leader_election.cc:343] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e300568403734babab4e5d9b6581178a (127.5.85.66:39545): Illegal state: must be running to vote when last-logged opid is not known
W20250625 07:59:58.501719 11907 leader_election.cc:343] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509): Illegal state: must be running to vote when last-logged opid is not known
I20250625 07:59:58.502056 11907 leader_election.cc:304] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 037821f13d3e4a039a8729a1256307fe; no voters: e300568403734babab4e5d9b6581178a, f39d2c8665264ac1a266072963a58329
I20250625 07:59:58.502815 12300 raft_consensus.cc:2747] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250625 07:59:58.670974 12045 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 07:59:58.671836 12045 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Bootstrap complete.
I20250625 07:59:58.673023 12045 ts_tablet_manager.cc:1397] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Time spent bootstrapping tablet: real 2.521s user 2.404s sys 0.080s
I20250625 07:59:58.678537 12045 raft_consensus.cc:357] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:58.680783 12045 raft_consensus.cc:738] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e300568403734babab4e5d9b6581178a, State: Initialized, Role: FOLLOWER
I20250625 07:59:58.681565 12045 consensus_queue.cc:260] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 07:59:58.684742 12045 ts_tablet_manager.cc:1428] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
I20250625 07:59:59.253674 12288 heartbeater.cc:499] Master 127.5.85.126:37951 was elected leader, sending a full tablet report...
I20250625 08:00:00.121914 12306 raft_consensus.cc:491] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:00:00.122385 12306 raft_consensus.cc:513] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:00.124783 12306 leader_election.cc:290] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509), 037821f13d3e4a039a8729a1256307fe (127.5.85.65:32813)
I20250625 08:00:00.150552 12243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7acda0f994e045acb1a45aafb8c890f1" candidate_uuid: "e300568403734babab4e5d9b6581178a" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f39d2c8665264ac1a266072963a58329" is_pre_election: true
W20250625 08:00:00.151836 12042 leader_election.cc:343] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:00:00.152823 11972 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7acda0f994e045acb1a45aafb8c890f1" candidate_uuid: "e300568403734babab4e5d9b6581178a" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "037821f13d3e4a039a8729a1256307fe" is_pre_election: true
I20250625 08:00:00.153429 11972 raft_consensus.cc:2466] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e300568403734babab4e5d9b6581178a in term 1.
I20250625 08:00:00.154985 12040 leader_election.cc:304] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 037821f13d3e4a039a8729a1256307fe, e300568403734babab4e5d9b6581178a; no voters: f39d2c8665264ac1a266072963a58329
I20250625 08:00:00.155691 12306 raft_consensus.cc:2802] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250625 08:00:00.156033 12306 raft_consensus.cc:491] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:00:00.156289 12306 raft_consensus.cc:3058] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Advancing to term 2
I20250625 08:00:00.162791 12306 raft_consensus.cc:513] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:00.164240 12306 leader_election.cc:290] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [CANDIDATE]: Term 2 election: Requested vote from peers f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509), 037821f13d3e4a039a8729a1256307fe (127.5.85.65:32813)
I20250625 08:00:00.165052 12243 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7acda0f994e045acb1a45aafb8c890f1" candidate_uuid: "e300568403734babab4e5d9b6581178a" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f39d2c8665264ac1a266072963a58329"
I20250625 08:00:00.165594 11972 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7acda0f994e045acb1a45aafb8c890f1" candidate_uuid: "e300568403734babab4e5d9b6581178a" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "037821f13d3e4a039a8729a1256307fe"
I20250625 08:00:00.166128 11972 raft_consensus.cc:3058] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Advancing to term 2
W20250625 08:00:00.166082 12042 leader_election.cc:343] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509): Illegal state: must be running to vote when last-logged opid is not known
I20250625 08:00:00.172173 11972 raft_consensus.cc:2466] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e300568403734babab4e5d9b6581178a in term 2.
I20250625 08:00:00.173051 12040 leader_election.cc:304] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 037821f13d3e4a039a8729a1256307fe, e300568403734babab4e5d9b6581178a; no voters: f39d2c8665264ac1a266072963a58329
I20250625 08:00:00.173640 12306 raft_consensus.cc:2802] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 2 FOLLOWER]: Leader election won for term 2
I20250625 08:00:00.175315 12306 raft_consensus.cc:695] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 2 LEADER]: Becoming Leader. State: Replica: e300568403734babab4e5d9b6581178a, State: Running, Role: LEADER
I20250625 08:00:00.176189 12306 consensus_queue.cc:237] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:00.189105 11828 catalog_manager.cc:5582] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a reported cstate change: term changed from 0 to 2, leader changed from <none> to e300568403734babab4e5d9b6581178a (127.5.85.66), VOTER 037821f13d3e4a039a8729a1256307fe (127.5.85.65) added, VOTER e300568403734babab4e5d9b6581178a (127.5.85.66) added, VOTER f39d2c8665264ac1a266072963a58329 (127.5.85.67) added. New cstate: current_term: 2 leader_uuid: "e300568403734babab4e5d9b6581178a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } health_report { overall_health: HEALTHY } } }
I20250625 08:00:00.346980 12181 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250625 08:00:00.347762 12181 tablet_bootstrap.cc:492] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Bootstrap complete.
I20250625 08:00:00.348920 12181 ts_tablet_manager.cc:1397] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Time spent bootstrapping tablet: real 2.333s user 2.222s sys 0.081s
I20250625 08:00:00.354182 12181 raft_consensus.cc:357] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:00.356395 12181 raft_consensus.cc:738] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39d2c8665264ac1a266072963a58329, State: Initialized, Role: FOLLOWER
I20250625 08:00:00.357199 12181 consensus_queue.cc:260] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:00.360239 12181 ts_tablet_manager.cc:1428] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20250625 08:00:00.568719 12243 raft_consensus.cc:3058] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 1 FOLLOWER]: Advancing to term 2
I20250625 08:00:00.579047 12243 raft_consensus.cc:1273] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 2 FOLLOWER]: Refusing update from remote peer e300568403734babab4e5d9b6581178a: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250625 08:00:00.580900 12306 consensus_queue.cc:1035] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
I20250625 08:00:00.599895 11972 raft_consensus.cc:1273] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 2 FOLLOWER]: Refusing update from remote peer e300568403734babab4e5d9b6581178a: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250625 08:00:00.601766 12311 consensus_queue.cc:1035] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [LEADER]: Connected to new peer: Peer: permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
I20250625 08:00:00.640614 12107 consensus_queue.cc:237] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:00.645478 12243 raft_consensus.cc:1273] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 2 FOLLOWER]: Refusing update from remote peer e300568403734babab4e5d9b6581178a: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250625 08:00:00.646978 12311 consensus_queue.cc:1035] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [LEADER]: Connected to new peer: Peer: permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250625 08:00:00.654781 12306 raft_consensus.cc:2953] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 037821f13d3e4a039a8729a1256307fe (127.5.85.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } } }
I20250625 08:00:00.657516 12242 raft_consensus.cc:2953] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 037821f13d3e4a039a8729a1256307fe (127.5.85.65) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } } }
I20250625 08:00:00.664847 11815 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 7acda0f994e045acb1a45aafb8c890f1 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 08:00:00.670679 11828 catalog_manager.cc:5582] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a reported cstate change: config changed from index -1 to 207, VOTER 037821f13d3e4a039a8729a1256307fe (127.5.85.65) evicted. New cstate: current_term: 2 leader_uuid: "e300568403734babab4e5d9b6581178a" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } health_report { overall_health: HEALTHY } } }
I20250625 08:00:00.683749 12107 consensus_queue.cc:237] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:00.686152 12311 raft_consensus.cc:2953] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER f39d2c8665264ac1a266072963a58329 (127.5.85.67) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } } }
W20250625 08:00:00.695394 5461 scanner-internal.cc:458] Time spent opening tablet: real 2.396s user 0.008s sys 0.000s
I20250625 08:00:00.698915 11815 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 7acda0f994e045acb1a45aafb8c890f1 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250625 08:00:00.703828 11828 catalog_manager.cc:5582] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a reported cstate change: config changed from index 207 to 208, VOTER f39d2c8665264ac1a266072963a58329 (127.5.85.67) evicted. New cstate: current_term: 2 leader_uuid: "e300568403734babab4e5d9b6581178a" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } health_report { overall_health: HEALTHY } } }
I20250625 08:00:00.704538 11952 tablet_service.cc:1515] Processing DeleteTablet for tablet 7acda0f994e045acb1a45aafb8c890f1 with delete_type TABLET_DATA_TOMBSTONED (TS 037821f13d3e4a039a8729a1256307fe not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:51188
I20250625 08:00:00.717387 12330 tablet_replica.cc:331] stopping tablet replica
I20250625 08:00:00.718523 12330 raft_consensus.cc:2241] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20250625 08:00:00.719334 12330 raft_consensus.cc:2270] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 08:00:00.750398 12330 ts_tablet_manager.cc:1905] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250625 08:00:00.748392 12223 tablet_service.cc:1515] Processing DeleteTablet for tablet 7acda0f994e045acb1a45aafb8c890f1 with delete_type TABLET_DATA_TOMBSTONED (TS f39d2c8665264ac1a266072963a58329 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:34862
I20250625 08:00:00.762668 12332 tablet_replica.cc:331] stopping tablet replica
I20250625 08:00:00.763705 12332 raft_consensus.cc:2241] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250625 08:00:00.764398 12332 raft_consensus.cc:2270] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 08:00:00.766266 12330 ts_tablet_manager.cc:1918] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.206
I20250625 08:00:00.766764 12330 log.cc:1199] T 7acda0f994e045acb1a45aafb8c890f1 P 037821f13d3e4a039a8729a1256307fe: Deleting WAL directory at /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/wal/wals/7acda0f994e045acb1a45aafb8c890f1
I20250625 08:00:00.768749 11814 catalog_manager.cc:4928] TS 037821f13d3e4a039a8729a1256307fe (127.5.85.65:32813): tablet 7acda0f994e045acb1a45aafb8c890f1 (table pre_rebuild [id=fdb6782a411740f089736018542bf1c9]) successfully deleted
I20250625 08:00:00.786585 12332 ts_tablet_manager.cc:1905] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250625 08:00:00.797338 12332 ts_tablet_manager.cc:1918] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250625 08:00:00.797883 12332 log.cc:1199] T 7acda0f994e045acb1a45aafb8c890f1 P f39d2c8665264ac1a266072963a58329: Deleting WAL directory at /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/wal/wals/7acda0f994e045acb1a45aafb8c890f1
I20250625 08:00:00.799381 11816 catalog_manager.cc:4928] TS f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509): tablet 7acda0f994e045acb1a45aafb8c890f1 (table pre_rebuild [id=fdb6782a411740f089736018542bf1c9]) successfully deleted
I20250625 08:00:01.383400 11952 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 08:00:01.440373 12087 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 08:00:01.447795 12223 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b024f4bf98434878977ea6818d7f9063 | 127.5.85.126:37951 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.5.85.84:41631 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
037821f13d3e4a039a8729a1256307fe | 127.5.85.65:32813 | HEALTHY | <none> | 0 | 0
e300568403734babab4e5d9b6581178a | 127.5.85.66:39545 | HEALTHY | <none> | 1 | 0
f39d2c8665264ac1a266072963a58329 | 127.5.85.67:39509 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.85.65 | experimental | 127.5.85.65:32813
local_ip_for_outbound_sockets | 127.5.85.66 | experimental | 127.5.85.66:39545
local_ip_for_outbound_sockets | 127.5.85.67 | experimental | 127.5.85.67:39509
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/info.pb | hidden | 127.5.85.65:32813
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/info.pb | hidden | 127.5.85.66:39545
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/info.pb | hidden | 127.5.85.67:39509
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.5.85.84:41631 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250625 08:00:01.719220 5461 log_verifier.cc:126] Checking tablet 7acda0f994e045acb1a45aafb8c890f1
I20250625 08:00:02.003427 5461 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 7acda0f994e045acb1a45aafb8c890f1
I20250625 08:00:02.005760 11829 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37748:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250625 08:00:02.006260 11829 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37748:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250625 08:00:02.018406 11829 catalog_manager.cc:5869] T 00000000000000000000000000000000 P b024f4bf98434878977ea6818d7f9063: Sending DeleteTablet for 1 replicas of tablet 7acda0f994e045acb1a45aafb8c890f1
I20250625 08:00:02.020099 5461 test_util.cc:276] Using random seed: -721227034
I20250625 08:00:02.020115 12087 tablet_service.cc:1515] Processing DeleteTablet for tablet 7acda0f994e045acb1a45aafb8c890f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-06-25 08:00:02 UTC) from {username='slave'} at 127.0.0.1:46934
I20250625 08:00:02.021857 12364 tablet_replica.cc:331] stopping tablet replica
I20250625 08:00:02.022614 12364 raft_consensus.cc:2241] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 2 LEADER]: Raft consensus shutting down.
I20250625 08:00:02.023289 12364 raft_consensus.cc:2270] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a [term 2 FOLLOWER]: Raft consensus is shut down!
I20250625 08:00:02.055756 11829 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58386:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250625 08:00:02.056849 12364 ts_tablet_manager.cc:1905] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250625 08:00:02.059628 11829 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250625 08:00:02.069065 12364 ts_tablet_manager.cc:1918] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250625 08:00:02.069543 12364 log.cc:1199] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Deleting WAL directory at /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/wal/wals/7acda0f994e045acb1a45aafb8c890f1
I20250625 08:00:02.070246 12364 ts_tablet_manager.cc:1939] T 7acda0f994e045acb1a45aafb8c890f1 P e300568403734babab4e5d9b6581178a: Deleting consensus metadata
I20250625 08:00:02.073628 11815 catalog_manager.cc:4928] TS e300568403734babab4e5d9b6581178a (127.5.85.66:39545): tablet 7acda0f994e045acb1a45aafb8c890f1 (table pre_rebuild [id=fdb6782a411740f089736018542bf1c9]) successfully deleted
I20250625 08:00:02.087635 11952 tablet_service.cc:1468] Processing CreateTablet for tablet 2c862f6b7a3f43a2aee5b2bdb1e006cb (DEFAULT_TABLE table=post_rebuild [id=02552d65bc304257a930c4a76faae823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:00:02.088246 12223 tablet_service.cc:1468] Processing CreateTablet for tablet 2c862f6b7a3f43a2aee5b2bdb1e006cb (DEFAULT_TABLE table=post_rebuild [id=02552d65bc304257a930c4a76faae823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:00:02.088836 11952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c862f6b7a3f43a2aee5b2bdb1e006cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:00:02.089619 12223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c862f6b7a3f43a2aee5b2bdb1e006cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:00:02.097226 12087 tablet_service.cc:1468] Processing CreateTablet for tablet 2c862f6b7a3f43a2aee5b2bdb1e006cb (DEFAULT_TABLE table=post_rebuild [id=02552d65bc304257a930c4a76faae823]), partition=RANGE (key) PARTITION UNBOUNDED
I20250625 08:00:02.098701 12087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c862f6b7a3f43a2aee5b2bdb1e006cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250625 08:00:02.108428 12372 tablet_bootstrap.cc:492] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329: Bootstrap starting.
I20250625 08:00:02.113008 12371 tablet_bootstrap.cc:492] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe: Bootstrap starting.
I20250625 08:00:02.117192 12372 tablet_bootstrap.cc:654] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329: Neither blocks nor log segments found. Creating new log.
I20250625 08:00:02.118831 12373 tablet_bootstrap.cc:492] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a: Bootstrap starting.
I20250625 08:00:02.123641 12371 tablet_bootstrap.cc:654] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe: Neither blocks nor log segments found. Creating new log.
I20250625 08:00:02.124996 12373 tablet_bootstrap.cc:654] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a: Neither blocks nor log segments found. Creating new log.
I20250625 08:00:02.132706 12371 tablet_bootstrap.cc:492] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe: No bootstrap required, opened a new log
I20250625 08:00:02.133255 12371 ts_tablet_manager.cc:1397] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.005s
I20250625 08:00:02.136092 12371 raft_consensus.cc:357] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:02.138541 12372 tablet_bootstrap.cc:492] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329: No bootstrap required, opened a new log
I20250625 08:00:02.138881 12371 raft_consensus.cc:383] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:00:02.139087 12372 ts_tablet_manager.cc:1397] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329: Time spent bootstrapping tablet: real 0.031s user 0.015s sys 0.000s
I20250625 08:00:02.139333 12371 raft_consensus.cc:738] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 037821f13d3e4a039a8729a1256307fe, State: Initialized, Role: FOLLOWER
I20250625 08:00:02.140336 12371 consensus_queue.cc:260] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:02.142226 12372 raft_consensus.cc:357] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:02.143133 12372 raft_consensus.cc:383] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:00:02.143497 12372 raft_consensus.cc:738] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f39d2c8665264ac1a266072963a58329, State: Initialized, Role: FOLLOWER
I20250625 08:00:02.144407 12372 consensus_queue.cc:260] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:02.145263 12373 tablet_bootstrap.cc:492] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a: No bootstrap required, opened a new log
I20250625 08:00:02.145762 12373 ts_tablet_manager.cc:1397] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.019s
I20250625 08:00:02.148738 12373 raft_consensus.cc:357] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:02.149626 12373 raft_consensus.cc:383] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250625 08:00:02.149955 12371 ts_tablet_manager.cc:1428] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe: Time spent starting tablet: real 0.016s user 0.007s sys 0.003s
I20250625 08:00:02.150008 12373 raft_consensus.cc:738] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e300568403734babab4e5d9b6581178a, State: Initialized, Role: FOLLOWER
I20250625 08:00:02.150926 12373 consensus_queue.cc:260] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:02.151588 12372 ts_tablet_manager.cc:1428] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329: Time spent starting tablet: real 0.012s user 0.005s sys 0.004s
I20250625 08:00:02.160624 12373 ts_tablet_manager.cc:1428] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a: Time spent starting tablet: real 0.014s user 0.010s sys 0.002s
I20250625 08:00:02.161865 12377 raft_consensus.cc:491] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250625 08:00:02.162355 12377 raft_consensus.cc:513] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:02.165294 12242 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c862f6b7a3f43a2aee5b2bdb1e006cb" candidate_uuid: "037821f13d3e4a039a8729a1256307fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39d2c8665264ac1a266072963a58329" is_pre_election: true
I20250625 08:00:02.165633 12107 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c862f6b7a3f43a2aee5b2bdb1e006cb" candidate_uuid: "037821f13d3e4a039a8729a1256307fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e300568403734babab4e5d9b6581178a" is_pre_election: true
I20250625 08:00:02.166163 12242 raft_consensus.cc:2466] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 037821f13d3e4a039a8729a1256307fe in term 0.
I20250625 08:00:02.166239 12107 raft_consensus.cc:2466] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 037821f13d3e4a039a8729a1256307fe in term 0.
I20250625 08:00:02.165565 12377 leader_election.cc:290] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509), e300568403734babab4e5d9b6581178a (127.5.85.66:39545)
I20250625 08:00:02.167454 11906 leader_election.cc:304] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037821f13d3e4a039a8729a1256307fe, e300568403734babab4e5d9b6581178a; no voters:
W20250625 08:00:02.168537 12153 tablet.cc:2378] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 08:00:02.169622 12377 raft_consensus.cc:2802] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250625 08:00:02.170048 12377 raft_consensus.cc:491] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250625 08:00:02.170434 12377 raft_consensus.cc:3058] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:00:02.177193 12377 raft_consensus.cc:513] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:02.179013 12377 leader_election.cc:290] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [CANDIDATE]: Term 1 election: Requested vote from peers f39d2c8665264ac1a266072963a58329 (127.5.85.67:39509), e300568403734babab4e5d9b6581178a (127.5.85.66:39545)
I20250625 08:00:02.179782 12242 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c862f6b7a3f43a2aee5b2bdb1e006cb" candidate_uuid: "037821f13d3e4a039a8729a1256307fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f39d2c8665264ac1a266072963a58329"
I20250625 08:00:02.180130 12107 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c862f6b7a3f43a2aee5b2bdb1e006cb" candidate_uuid: "037821f13d3e4a039a8729a1256307fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e300568403734babab4e5d9b6581178a"
I20250625 08:00:02.180367 12242 raft_consensus.cc:3058] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329 [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:00:02.180682 12107 raft_consensus.cc:3058] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a [term 0 FOLLOWER]: Advancing to term 1
I20250625 08:00:02.187636 12107 raft_consensus.cc:2466] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 037821f13d3e4a039a8729a1256307fe in term 1.
I20250625 08:00:02.188901 11906 leader_election.cc:304] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 037821f13d3e4a039a8729a1256307fe, e300568403734babab4e5d9b6581178a; no voters:
I20250625 08:00:02.191190 12242 raft_consensus.cc:2466] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 037821f13d3e4a039a8729a1256307fe in term 1.
I20250625 08:00:02.192260 12377 raft_consensus.cc:2802] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 1 FOLLOWER]: Leader election won for term 1
I20250625 08:00:02.193462 12377 raft_consensus.cc:695] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [term 1 LEADER]: Becoming Leader. State: Replica: 037821f13d3e4a039a8729a1256307fe, State: Running, Role: LEADER
I20250625 08:00:02.194391 12377 consensus_queue.cc:237] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } } peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } }
I20250625 08:00:02.203886 11829 catalog_manager.cc:5582] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 037821f13d3e4a039a8729a1256307fe (127.5.85.65). New cstate: current_term: 1 leader_uuid: "037821f13d3e4a039a8729a1256307fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "037821f13d3e4a039a8729a1256307fe" member_type: VOTER last_known_addr { host: "127.5.85.65" port: 32813 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 } health_report { overall_health: UNKNOWN } } }
W20250625 08:00:02.241374 12289 tablet.cc:2378] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250625 08:00:02.286804 12018 tablet.cc:2378] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250625 08:00:02.334086 12107 raft_consensus.cc:1273] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P e300568403734babab4e5d9b6581178a [term 1 FOLLOWER]: Refusing update from remote peer 037821f13d3e4a039a8729a1256307fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 08:00:02.334194 12242 raft_consensus.cc:1273] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P f39d2c8665264ac1a266072963a58329 [term 1 FOLLOWER]: Refusing update from remote peer 037821f13d3e4a039a8729a1256307fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250625 08:00:02.335894 12377 consensus_queue.cc:1035] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "e300568403734babab4e5d9b6581178a" member_type: VOTER last_known_addr { host: "127.5.85.66" port: 39545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250625 08:00:02.342993 12380 consensus_queue.cc:1035] T 2c862f6b7a3f43a2aee5b2bdb1e006cb P 037821f13d3e4a039a8729a1256307fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "f39d2c8665264ac1a266072963a58329" member_type: VOTER last_known_addr { host: "127.5.85.67" port: 39509 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250625 08:00:02.380445 12388 mvcc.cc:204] Tried to move back new op lower bound from 7171434095943692288 to 7171434095398019072. Current Snapshot: MvccSnapshot[applied={T|T < 7171434095943692288}]
I20250625 08:00:02.385932 12390 mvcc.cc:204] Tried to move back new op lower bound from 7171434095943692288 to 7171434095398019072. Current Snapshot: MvccSnapshot[applied={T|T < 7171434095943692288}]
W20250625 08:00:03.171718 12384 meta_cache.cc:1261] Time spent looking up entry by key: real 0.061s user 0.004s sys 0.000s
I20250625 08:00:07.485314 11952 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250625 08:00:07.488137 12087 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250625 08:00:07.491029 12223 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+--------------------+---------
b024f4bf98434878977ea6818d7f9063 | 127.5.85.126:37951 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.5.85.84:41631 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+-------------------+---------+----------+----------------+-----------------
037821f13d3e4a039a8729a1256307fe | 127.5.85.65:32813 | HEALTHY | <none> | 1 | 0
e300568403734babab4e5d9b6581178a | 127.5.85.66:39545 | HEALTHY | <none> | 0 | 0
f39d2c8665264ac1a266072963a58329 | 127.5.85.67:39509 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.5.85.65 | experimental | 127.5.85.65:32813
local_ip_for_outbound_sockets | 127.5.85.66 | experimental | 127.5.85.66:39545
local_ip_for_outbound_sockets | 127.5.85.67 | experimental | 127.5.85.67:39509
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-0/data/info.pb | hidden | 127.5.85.65:32813
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-1/data/info.pb | hidden | 127.5.85.66:39545
server_dump_info_path | /tmp/dist-test-taskBp1FTP/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1750838232495789-5461-0/minicluster-data/ts-2/data/info.pb | hidden | 127.5.85.67:39509
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+-------------------+-------------------------
builtin_ntp_servers | 127.5.85.84:41631 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250625 08:00:07.730832 5461 log_verifier.cc:126] Checking tablet 2c862f6b7a3f43a2aee5b2bdb1e006cb
I20250625 08:00:08.476800 5461 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 2c862f6b7a3f43a2aee5b2bdb1e006cb
I20250625 08:00:08.477674 5461 log_verifier.cc:126] Checking tablet 7acda0f994e045acb1a45aafb8c890f1
I20250625 08:00:08.477962 5461 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 7acda0f994e045acb1a45aafb8c890f1
I20250625 08:00:08.502427 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 11866
I20250625 08:00:08.537093 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 12021
I20250625 08:00:08.570289 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 12156
I20250625 08:00:08.601841 5461 external_mini_cluster.cc:1620] Killing /tmp/dist-test-taskBp1FTP/build/tsan/bin/kudu with pid 11796
2025-06-25T08:00:08Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (34316 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (34316 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (176089 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST