Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250810 20:27:28.929418 31695 test_util.cc:276] Using random seed: -1563707010
W20250810 20:27:30.215042 31695 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.234s user 0.000s sys 0.003s
W20250810 20:27:30.215376 31695 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.235s user 0.000s sys 0.006s
I20250810 20:27:30.218382 31695 ts_itest-base.cc:115] Starting cluster with:
I20250810 20:27:30.218575 31695 ts_itest-base.cc:116] --------------
I20250810 20:27:30.218750 31695 ts_itest-base.cc:117] 4 tablet servers
I20250810 20:27:30.218961 31695 ts_itest-base.cc:118] 3 replicas per TS
I20250810 20:27:30.219141 31695 ts_itest-base.cc:119] --------------
2025-08-10T20:27:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T20:27:30Z Disabled control of system clock
I20250810 20:27:30.263160 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:39215
--webserver_interface=127.30.243.254
--webserver_port=0
--builtin_ntp_servers=127.30.243.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:39215 with env {}
W20250810 20:27:30.600277 31709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:30.600912 31709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:27:30.601403 31709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:30.632915 31709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:27:30.633312 31709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:27:30.633575 31709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:27:30.633822 31709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:27:30.669618 31709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:39215
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:39215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:27:30.670992 31709 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:27:30.672614 31709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:27:30.686178 31715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:30.687649 31716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:31.919005 31718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:31.923561 31717 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1234 milliseconds
I20250810 20:27:31.923676 31709 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:27:31.924912 31709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:27:31.928119 31709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:27:31.929570 31709 hybrid_clock.cc:648] HybridClock initialized: now 1754857651929525 us; error 60 us; skew 500 ppm
I20250810 20:27:31.930380 31709 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:27:31.938143 31709 webserver.cc:489] Webserver started at http://127.30.243.254:34593/ using document root <none> and password file <none>
I20250810 20:27:31.939148 31709 fs_manager.cc:362] Metadata directory not provided
I20250810 20:27:31.939347 31709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:27:31.939817 31709 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:27:31.944361 31709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492"
format_stamp: "Formatted at 2025-08-10 20:27:31 on dist-test-slave-7swh"
I20250810 20:27:31.945472 31709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492"
format_stamp: "Formatted at 2025-08-10 20:27:31 on dist-test-slave-7swh"
I20250810 20:27:31.953400 31709 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250810 20:27:31.959692 31725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:31.960794 31709 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250810 20:27:31.961151 31709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492"
format_stamp: "Formatted at 2025-08-10 20:27:31 on dist-test-slave-7swh"
I20250810 20:27:31.961468 31709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:27:32.008689 31709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:27:32.010401 31709 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:27:32.010859 31709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:27:32.102382 31709 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:39215
I20250810 20:27:32.102454 31776 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:39215 every 8 connection(s)
I20250810 20:27:32.105140 31709 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 20:27:32.111887 31777 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:27:32.111923 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 31709
I20250810 20:27:32.112408 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 20:27:32.134423 31777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492: Bootstrap starting.
I20250810 20:27:32.140846 31777 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492: Neither blocks nor log segments found. Creating new log.
I20250810 20:27:32.143404 31777 log.cc:826] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492: Log is configured to *not* fsync() on all Append() calls
I20250810 20:27:32.149859 31777 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492: No bootstrap required, opened a new log
I20250810 20:27:32.172304 31777 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } }
I20250810 20:27:32.173171 31777 raft_consensus.cc:383] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:27:32.173480 31777 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbacf430d3f4c22be9fbbe4fc1a3492, State: Initialized, Role: FOLLOWER
I20250810 20:27:32.174414 31777 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } }
I20250810 20:27:32.175134 31777 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:27:32.175384 31777 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:27:32.175670 31777 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:27:32.181504 31777 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } }
I20250810 20:27:32.182266 31777 leader_election.cc:304] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fbacf430d3f4c22be9fbbe4fc1a3492; no voters:
I20250810 20:27:32.184162 31777 leader_election.cc:290] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:27:32.185514 31782 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:27:32.188995 31782 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 1 LEADER]: Becoming Leader. State: Replica: 1fbacf430d3f4c22be9fbbe4fc1a3492, State: Running, Role: LEADER
I20250810 20:27:32.189335 31777 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:27:32.189817 31782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } }
I20250810 20:27:32.204948 31783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } } }
I20250810 20:27:32.205873 31783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [sys.catalog]: This master's current role is: LEADER
I20250810 20:27:32.207363 31784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fbacf430d3f4c22be9fbbe4fc1a3492. Latest consensus state: current_term: 1 leader_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } } }
I20250810 20:27:32.208139 31784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [sys.catalog]: This master's current role is: LEADER
I20250810 20:27:32.214331 31791 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:27:32.227916 31791 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:27:32.249933 31791 catalog_manager.cc:1349] Generated new cluster ID: 5eca0178bcd2423ca9dbd3afdc54c736
I20250810 20:27:32.250274 31791 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:27:32.273202 31791 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 20:27:32.274765 31791 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:27:32.294659 31791 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492: Generated new TSK 0
I20250810 20:27:32.295588 31791 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 20:27:32.321686 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:0
--local_ip_for_outbound_sockets=127.30.243.193
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:39215
--builtin_ntp_servers=127.30.243.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250810 20:27:32.691736 31802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:32.692273 31802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:27:32.692817 31802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:32.725672 31802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:27:32.726547 31802 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:27:32.762673 31802 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:39215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:27:32.764081 31802 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:27:32.765811 31802 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:27:32.781101 31808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:34.183635 31807 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 31802
W20250810 20:27:34.318778 31802 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.536s user 0.514s sys 1.019s
W20250810 20:27:34.320430 31802 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.538s user 0.514s sys 1.019s
W20250810 20:27:32.782716 31809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:34.321552 31810 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1536 milliseconds
W20250810 20:27:34.322059 31811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:27:34.322058 31802 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:27:34.326300 31802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:27:34.329094 31802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:27:34.330634 31802 hybrid_clock.cc:648] HybridClock initialized: now 1754857654330585 us; error 52 us; skew 500 ppm
I20250810 20:27:34.331760 31802 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:27:34.341080 31802 webserver.cc:489] Webserver started at http://127.30.243.193:38531/ using document root <none> and password file <none>
I20250810 20:27:34.342517 31802 fs_manager.cc:362] Metadata directory not provided
I20250810 20:27:34.342878 31802 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:27:34.343505 31802 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:27:34.350188 31802 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "a7f85b12af694793950a560b471ad9de"
format_stamp: "Formatted at 2025-08-10 20:27:34 on dist-test-slave-7swh"
I20250810 20:27:34.351985 31802 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "a7f85b12af694793950a560b471ad9de"
format_stamp: "Formatted at 2025-08-10 20:27:34 on dist-test-slave-7swh"
I20250810 20:27:34.361871 31802 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s
I20250810 20:27:34.370370 31818 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:34.371712 31802 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.003s
I20250810 20:27:34.372149 31802 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "a7f85b12af694793950a560b471ad9de"
format_stamp: "Formatted at 2025-08-10 20:27:34 on dist-test-slave-7swh"
I20250810 20:27:34.372598 31802 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:27:34.439327 31802 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:27:34.440912 31802 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:27:34.441402 31802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:27:34.445231 31802 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:27:34.450325 31802 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:27:34.450558 31802 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:34.450842 31802 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:27:34.451049 31802 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250810 20:27:34.677981 31802 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:43391
I20250810 20:27:34.678148 31930 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:43391 every 8 connection(s)
I20250810 20:27:34.680631 31802 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 20:27:34.686182 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 31802
I20250810 20:27:34.686695 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 20:27:34.694770 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.194:0
--local_ip_for_outbound_sockets=127.30.243.194
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:39215
--builtin_ntp_servers=127.30.243.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:27:34.710969 31931 heartbeater.cc:344] Connected to a master server at 127.30.243.254:39215
I20250810 20:27:34.711524 31931 heartbeater.cc:461] Registering TS with master...
I20250810 20:27:34.712915 31931 heartbeater.cc:507] Master 127.30.243.254:39215 requested a full tablet report, sending...
I20250810 20:27:34.716491 31742 ts_manager.cc:194] Registered new tserver with Master: a7f85b12af694793950a560b471ad9de (127.30.243.193:43391)
I20250810 20:27:34.719898 31742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:46333
W20250810 20:27:35.042627 31935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:35.043191 31935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:27:35.043710 31935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:35.075773 31935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:27:35.076668 31935 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.194
I20250810 20:27:35.113966 31935 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:39215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.194
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:27:35.115427 31935 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:27:35.117136 31935 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:27:35.131253 31941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:27:35.725768 31931 heartbeater.cc:499] Master 127.30.243.254:39215 was elected leader, sending a full tablet report...
W20250810 20:27:35.134320 31942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:35.139074 31944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:36.417135 31943 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1280 milliseconds
I20250810 20:27:36.417272 31935 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:27:36.418567 31935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:27:36.421512 31935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:27:36.423094 31935 hybrid_clock.cc:648] HybridClock initialized: now 1754857656423030 us; error 72 us; skew 500 ppm
I20250810 20:27:36.423925 31935 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:27:36.431809 31935 webserver.cc:489] Webserver started at http://127.30.243.194:36943/ using document root <none> and password file <none>
I20250810 20:27:36.432802 31935 fs_manager.cc:362] Metadata directory not provided
I20250810 20:27:36.433058 31935 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:27:36.433669 31935 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:27:36.438923 31935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f5be77523b4840c8a20a763a423b75c0"
format_stamp: "Formatted at 2025-08-10 20:27:36 on dist-test-slave-7swh"
I20250810 20:27:36.440235 31935 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f5be77523b4840c8a20a763a423b75c0"
format_stamp: "Formatted at 2025-08-10 20:27:36 on dist-test-slave-7swh"
I20250810 20:27:36.448711 31935 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20250810 20:27:36.455155 31951 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:36.456336 31935 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 20:27:36.456789 31935 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f5be77523b4840c8a20a763a423b75c0"
format_stamp: "Formatted at 2025-08-10 20:27:36 on dist-test-slave-7swh"
I20250810 20:27:36.457159 31935 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:27:36.512903 31935 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:27:36.514544 31935 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:27:36.515049 31935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:27:36.518460 31935 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:27:36.523449 31935 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:27:36.523762 31935 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:36.524030 31935 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:27:36.524199 31935 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:36.727905 31935 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.194:43945
I20250810 20:27:36.728073 32063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.194:43945 every 8 connection(s)
I20250810 20:27:36.730612 31935 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 20:27:36.736759 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 31935
I20250810 20:27:36.737424 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 20:27:36.746901 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.195:0
--local_ip_for_outbound_sockets=127.30.243.195
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:39215
--builtin_ntp_servers=127.30.243.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:27:36.765039 32064 heartbeater.cc:344] Connected to a master server at 127.30.243.254:39215
I20250810 20:27:36.765518 32064 heartbeater.cc:461] Registering TS with master...
I20250810 20:27:36.766602 32064 heartbeater.cc:507] Master 127.30.243.254:39215 requested a full tablet report, sending...
I20250810 20:27:36.769083 31742 ts_manager.cc:194] Registered new tserver with Master: f5be77523b4840c8a20a763a423b75c0 (127.30.243.194:43945)
I20250810 20:27:36.770607 31742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.194:43591
W20250810 20:27:37.136049 32068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:37.136575 32068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:27:37.137094 32068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:37.169250 32068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:27:37.170151 32068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.195
I20250810 20:27:37.205210 32068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:39215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.195
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:27:37.206561 32068 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:27:37.208254 32068 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:27:37.222218 32074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:27:37.774873 32064 heartbeater.cc:499] Master 127.30.243.254:39215 was elected leader, sending a full tablet report...
W20250810 20:27:37.223474 32075 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:38.472965 32077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:38.476018 32076 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1247 milliseconds
I20250810 20:27:38.476147 32068 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:27:38.477409 32068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:27:38.480365 32068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:27:38.481884 32068 hybrid_clock.cc:648] HybridClock initialized: now 1754857658481802 us; error 93 us; skew 500 ppm
I20250810 20:27:38.482762 32068 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:27:38.492478 32068 webserver.cc:489] Webserver started at http://127.30.243.195:40831/ using document root <none> and password file <none>
I20250810 20:27:38.493585 32068 fs_manager.cc:362] Metadata directory not provided
I20250810 20:27:38.493835 32068 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:27:38.494366 32068 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:27:38.499100 32068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "724fa811237348079bf031f6b1e5057b"
format_stamp: "Formatted at 2025-08-10 20:27:38 on dist-test-slave-7swh"
I20250810 20:27:38.500432 32068 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "724fa811237348079bf031f6b1e5057b"
format_stamp: "Formatted at 2025-08-10 20:27:38 on dist-test-slave-7swh"
I20250810 20:27:38.508172 32068 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250810 20:27:38.514863 32084 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:38.516129 32068 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250810 20:27:38.516505 32068 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "724fa811237348079bf031f6b1e5057b"
format_stamp: "Formatted at 2025-08-10 20:27:38 on dist-test-slave-7swh"
I20250810 20:27:38.516960 32068 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:27:38.570695 32068 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:27:38.572441 32068 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:27:38.572925 32068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:27:38.576153 32068 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:27:38.580670 32068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:27:38.580900 32068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:38.581176 32068 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:27:38.581343 32068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:38.767149 32068 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.195:38463
I20250810 20:27:38.767230 32196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.195:38463 every 8 connection(s)
I20250810 20:27:38.769734 32068 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 20:27:38.775362 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 32068
I20250810 20:27:38.775771 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 20:27:38.783388 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.196:0
--local_ip_for_outbound_sockets=127.30.243.196
--webserver_interface=127.30.243.196
--webserver_port=0
--tserver_master_addrs=127.30.243.254:39215
--builtin_ntp_servers=127.30.243.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:27:38.803185 32197 heartbeater.cc:344] Connected to a master server at 127.30.243.254:39215
I20250810 20:27:38.803814 32197 heartbeater.cc:461] Registering TS with master...
I20250810 20:27:38.805482 32197 heartbeater.cc:507] Master 127.30.243.254:39215 requested a full tablet report, sending...
I20250810 20:27:38.808207 31742 ts_manager.cc:194] Registered new tserver with Master: 724fa811237348079bf031f6b1e5057b (127.30.243.195:38463)
I20250810 20:27:38.809453 31742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.195:41603
W20250810 20:27:39.119874 32201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:39.120429 32201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:27:39.120970 32201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:39.153985 32201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:27:39.154839 32201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.196
I20250810 20:27:39.190495 32201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.243.196
--webserver_port=0
--tserver_master_addrs=127.30.243.254:39215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.196
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:27:39.191905 32201 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:27:39.193555 32201 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:27:39.206429 32208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:27:39.813330 32197 heartbeater.cc:499] Master 127.30.243.254:39215 was elected leader, sending a full tablet report...
W20250810 20:27:39.208913 32209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:39.213750 32211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:40.413481 32210 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250810 20:27:40.413561 32201 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:27:40.417765 32201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:27:40.420395 32201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:27:40.421837 32201 hybrid_clock.cc:648] HybridClock initialized: now 1754857660421785 us; error 58 us; skew 500 ppm
I20250810 20:27:40.422658 32201 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:27:40.431123 32201 webserver.cc:489] Webserver started at http://127.30.243.196:43191/ using document root <none> and password file <none>
I20250810 20:27:40.432161 32201 fs_manager.cc:362] Metadata directory not provided
I20250810 20:27:40.432399 32201 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:27:40.432904 32201 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:27:40.437557 32201 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "b176880e3fe5469a8b3393604567e4d2"
format_stamp: "Formatted at 2025-08-10 20:27:40 on dist-test-slave-7swh"
I20250810 20:27:40.438748 32201 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "b176880e3fe5469a8b3393604567e4d2"
format_stamp: "Formatted at 2025-08-10 20:27:40 on dist-test-slave-7swh"
I20250810 20:27:40.447008 32201 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250810 20:27:40.453753 32218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:40.454982 32201 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20250810 20:27:40.455330 32201 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "b176880e3fe5469a8b3393604567e4d2"
format_stamp: "Formatted at 2025-08-10 20:27:40 on dist-test-slave-7swh"
I20250810 20:27:40.455675 32201 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:27:40.516348 32201 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:27:40.518106 32201 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:27:40.518699 32201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:27:40.522033 32201 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:27:40.526803 32201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:27:40.527060 32201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250810 20:27:40.527382 32201 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:27:40.527534 32201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:40.700801 32201 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.196:37121
I20250810 20:27:40.700874 32331 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.196:37121 every 8 connection(s)
I20250810 20:27:40.703519 32201 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250810 20:27:40.712608 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 32201
I20250810 20:27:40.713331 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250810 20:27:40.732645 32332 heartbeater.cc:344] Connected to a master server at 127.30.243.254:39215
I20250810 20:27:40.733327 32332 heartbeater.cc:461] Registering TS with master...
I20250810 20:27:40.734855 32332 heartbeater.cc:507] Master 127.30.243.254:39215 requested a full tablet report, sending...
I20250810 20:27:40.737668 31742 ts_manager.cc:194] Registered new tserver with Master: b176880e3fe5469a8b3393604567e4d2 (127.30.243.196:37121)
I20250810 20:27:40.739353 31742 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.196:35531
I20250810 20:27:40.752820 31695 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250810 20:27:40.806389 31742 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36566:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250810 20:27:40.901163 32267 tablet_service.cc:1468] Processing CreateTablet for tablet c22e4f3ebb6541d98c1c774c6b483731 (DEFAULT_TABLE table=TestTable [id=d8ab97a7372f48d0bc8a05746d4e17c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:27:40.904461 31999 tablet_service.cc:1468] Processing CreateTablet for tablet c22e4f3ebb6541d98c1c774c6b483731 (DEFAULT_TABLE table=TestTable [id=d8ab97a7372f48d0bc8a05746d4e17c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:27:40.905171 32267 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22e4f3ebb6541d98c1c774c6b483731. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:27:40.904569 31866 tablet_service.cc:1468] Processing CreateTablet for tablet c22e4f3ebb6541d98c1c774c6b483731 (DEFAULT_TABLE table=TestTable [id=d8ab97a7372f48d0bc8a05746d4e17c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:27:40.906514 31999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22e4f3ebb6541d98c1c774c6b483731. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:27:40.906919 31866 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22e4f3ebb6541d98c1c774c6b483731. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:27:40.935005 32351 tablet_bootstrap.cc:492] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0: Bootstrap starting.
I20250810 20:27:40.941136 32352 tablet_bootstrap.cc:492] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2: Bootstrap starting.
I20250810 20:27:40.943708 32353 tablet_bootstrap.cc:492] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de: Bootstrap starting.
I20250810 20:27:40.945611 32351 tablet_bootstrap.cc:654] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0: Neither blocks nor log segments found. Creating new log.
I20250810 20:27:40.950508 32351 log.cc:826] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0: Log is configured to *not* fsync() on all Append() calls
I20250810 20:27:40.951325 32352 tablet_bootstrap.cc:654] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2: Neither blocks nor log segments found. Creating new log.
I20250810 20:27:40.952196 32353 tablet_bootstrap.cc:654] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de: Neither blocks nor log segments found. Creating new log.
I20250810 20:27:40.954403 32352 log.cc:826] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2: Log is configured to *not* fsync() on all Append() calls
I20250810 20:27:40.955365 32353 log.cc:826] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de: Log is configured to *not* fsync() on all Append() calls
I20250810 20:27:40.958372 32351 tablet_bootstrap.cc:492] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0: No bootstrap required, opened a new log
I20250810 20:27:40.959075 32351 ts_tablet_manager.cc:1397] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.010s
I20250810 20:27:40.961336 32352 tablet_bootstrap.cc:492] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2: No bootstrap required, opened a new log
I20250810 20:27:40.961880 32352 ts_tablet_manager.cc:1397] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.016s
I20250810 20:27:40.966351 32353 tablet_bootstrap.cc:492] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de: No bootstrap required, opened a new log
I20250810 20:27:40.967095 32353 ts_tablet_manager.cc:1397] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.011s
I20250810 20:27:40.982460 32352 raft_consensus.cc:357] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:40.983271 32352 raft_consensus.cc:383] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:27:40.983610 32352 raft_consensus.cc:738] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b176880e3fe5469a8b3393604567e4d2, State: Initialized, Role: FOLLOWER
I20250810 20:27:40.984938 32352 consensus_queue.cc:260] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:40.989609 32332 heartbeater.cc:499] Master 127.30.243.254:39215 was elected leader, sending a full tablet report...
I20250810 20:27:40.991061 32352 ts_tablet_manager.cc:1428] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2: Time spent starting tablet: real 0.029s user 0.020s sys 0.009s
I20250810 20:27:40.992755 32351 raft_consensus.cc:357] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:40.993925 32351 raft_consensus.cc:383] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:27:40.994256 32351 raft_consensus.cc:738] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5be77523b4840c8a20a763a423b75c0, State: Initialized, Role: FOLLOWER
I20250810 20:27:40.995219 32351 consensus_queue.cc:260] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:40.999498 32353 raft_consensus.cc:357] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:41.001300 32353 raft_consensus.cc:383] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:27:41.001659 32353 raft_consensus.cc:738] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7f85b12af694793950a560b471ad9de, State: Initialized, Role: FOLLOWER
I20250810 20:27:41.001828 32351 ts_tablet_manager.cc:1428] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0: Time spent starting tablet: real 0.042s user 0.030s sys 0.008s
I20250810 20:27:41.002770 32353 consensus_queue.cc:260] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:41.006903 32353 ts_tablet_manager.cc:1428] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de: Time spent starting tablet: real 0.039s user 0.037s sys 0.001s
I20250810 20:27:41.012380 32357 raft_consensus.cc:491] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:27:41.013236 32357 raft_consensus.cc:513] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:41.016139 32357 leader_election.cc:290] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7f85b12af694793950a560b471ad9de (127.30.243.193:43391), f5be77523b4840c8a20a763a423b75c0 (127.30.243.194:43945)
I20250810 20:27:41.026214 31886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22e4f3ebb6541d98c1c774c6b483731" candidate_uuid: "b176880e3fe5469a8b3393604567e4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7f85b12af694793950a560b471ad9de" is_pre_election: true
I20250810 20:27:41.027491 31886 raft_consensus.cc:2466] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b176880e3fe5469a8b3393604567e4d2 in term 0.
I20250810 20:27:41.029047 32219 leader_election.cc:304] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7f85b12af694793950a560b471ad9de, b176880e3fe5469a8b3393604567e4d2; no voters:
I20250810 20:27:41.029784 32357 raft_consensus.cc:2802] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 20:27:41.030104 32357 raft_consensus.cc:491] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 20:27:41.030408 32357 raft_consensus.cc:3058] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:27:41.030177 32019 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22e4f3ebb6541d98c1c774c6b483731" candidate_uuid: "b176880e3fe5469a8b3393604567e4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5be77523b4840c8a20a763a423b75c0" is_pre_election: true
I20250810 20:27:41.031189 32019 raft_consensus.cc:2466] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b176880e3fe5469a8b3393604567e4d2 in term 0.
I20250810 20:27:41.035969 32357 raft_consensus.cc:513] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:41.037503 32357 leader_election.cc:290] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [CANDIDATE]: Term 1 election: Requested vote from peers a7f85b12af694793950a560b471ad9de (127.30.243.193:43391), f5be77523b4840c8a20a763a423b75c0 (127.30.243.194:43945)
I20250810 20:27:41.038434 31886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22e4f3ebb6541d98c1c774c6b483731" candidate_uuid: "b176880e3fe5469a8b3393604567e4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7f85b12af694793950a560b471ad9de"
I20250810 20:27:41.038493 32019 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22e4f3ebb6541d98c1c774c6b483731" candidate_uuid: "b176880e3fe5469a8b3393604567e4d2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5be77523b4840c8a20a763a423b75c0"
I20250810 20:27:41.039112 31886 raft_consensus.cc:3058] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:27:41.039112 32019 raft_consensus.cc:3058] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:27:41.046911 31886 raft_consensus.cc:2466] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b176880e3fe5469a8b3393604567e4d2 in term 1.
I20250810 20:27:41.047308 32019 raft_consensus.cc:2466] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b176880e3fe5469a8b3393604567e4d2 in term 1.
I20250810 20:27:41.048110 32219 leader_election.cc:304] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7f85b12af694793950a560b471ad9de, b176880e3fe5469a8b3393604567e4d2; no voters:
I20250810 20:27:41.048939 32357 raft_consensus.cc:2802] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:27:41.051203 32357 raft_consensus.cc:695] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [term 1 LEADER]: Becoming Leader. State: Replica: b176880e3fe5469a8b3393604567e4d2, State: Running, Role: LEADER
I20250810 20:27:41.052299 32357 consensus_queue.cc:237] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:41.064772 31740 catalog_manager.cc:5582] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 reported cstate change: term changed from 0 to 1, leader changed from <none> to b176880e3fe5469a8b3393604567e4d2 (127.30.243.196). New cstate: current_term: 1 leader_uuid: "b176880e3fe5469a8b3393604567e4d2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } health_report { overall_health: UNKNOWN } } }
I20250810 20:27:41.095161 31695 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250810 20:27:41.099818 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a7f85b12af694793950a560b471ad9de to finish bootstrapping
I20250810 20:27:41.119199 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f5be77523b4840c8a20a763a423b75c0 to finish bootstrapping
I20250810 20:27:41.131770 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b176880e3fe5469a8b3393604567e4d2 to finish bootstrapping
I20250810 20:27:41.144960 31695 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250810 20:27:41.149323 31695 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "c22e4f3ebb6541d98c1c774c6b483731"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "b176880e3fe5469a8b3393604567e4d2"
rpc_addresses {
host: "127.30.243.196"
port: 37121
}
}
ts_infos {
permanent_uuid: "a7f85b12af694793950a560b471ad9de"
rpc_addresses {
host: "127.30.243.193"
port: 43391
}
}
ts_infos {
permanent_uuid: "f5be77523b4840c8a20a763a423b75c0"
rpc_addresses {
host: "127.30.243.194"
port: 43945
}
}
W20250810 20:27:41.194365 31932 tablet.cc:2378] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 20:27:41.209829 32333 tablet.cc:2378] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 20:27:41.247892 32065 tablet.cc:2378] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:27:41.529111 32357 consensus_queue.cc:1035] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 20:27:41.546818 32361 consensus_queue.cc:1035] T c22e4f3ebb6541d98c1c774c6b483731 P b176880e3fe5469a8b3393604567e4d2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 20:27:41.552518 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 32201
W20250810 20:27:41.586016 31953 connection.cc:537] server connection from 127.30.243.196:38025 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250810 20:27:41.586643 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 31709
I20250810 20:27:41.612371 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:39215
--webserver_interface=127.30.243.254
--webserver_port=34593
--builtin_ntp_servers=127.30.243.212:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:39215 with env {}
W20250810 20:27:41.825624 32197 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:39215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:39215: connect: Connection refused (error 111)
W20250810 20:27:41.937273 32372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:41.937932 32372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:27:41.938429 32372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:41.971963 32372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:27:41.972312 32372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:27:41.972563 32372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:27:41.972805 32372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:27:42.011039 32372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:39215
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:39215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=34593
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:27:42.012584 32372 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:27:42.014439 32372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:27:42.026317 32379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:42.578097 31931 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:39215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:39215: connect: Connection refused (error 111)
W20250810 20:27:42.582262 32064 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:39215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:39215: connect: Connection refused (error 111)
I20250810 20:27:43.061116 32389 raft_consensus.cc:491] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b176880e3fe5469a8b3393604567e4d2)
I20250810 20:27:43.062747 32389 raft_consensus.cc:513] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
W20250810 20:27:43.071237 31820 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.243.196:37121: connect: Connection refused (error 111)
I20250810 20:27:43.072399 32390 raft_consensus.cc:491] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b176880e3fe5469a8b3393604567e4d2)
I20250810 20:27:43.073060 32390 raft_consensus.cc:513] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:43.077090 32390 leader_election.cc:290] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b176880e3fe5469a8b3393604567e4d2 (127.30.243.196:37121), a7f85b12af694793950a560b471ad9de (127.30.243.193:43391)
W20250810 20:27:43.079571 31953 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.243.196:37121: connect: Connection refused (error 111)
I20250810 20:27:43.103292 32389 leader_election.cc:290] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b176880e3fe5469a8b3393604567e4d2 (127.30.243.196:37121), f5be77523b4840c8a20a763a423b75c0 (127.30.243.194:43945)
W20250810 20:27:43.123337 31820 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.30.243.196:37121, user_credentials={real_user=slave}} blocked reactor thread for 52314.7us
W20250810 20:27:43.128103 31820 leader_election.cc:336] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b176880e3fe5469a8b3393604567e4d2 (127.30.243.196:37121): Network error: Client connection negotiation failed: client connection to 127.30.243.196:37121: connect: Connection refused (error 111)
I20250810 20:27:43.135535 32019 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22e4f3ebb6541d98c1c774c6b483731" candidate_uuid: "a7f85b12af694793950a560b471ad9de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f5be77523b4840c8a20a763a423b75c0" is_pre_election: true
I20250810 20:27:43.136466 32019 raft_consensus.cc:2466] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a7f85b12af694793950a560b471ad9de in term 1.
I20250810 20:27:43.138586 31820 leader_election.cc:304] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a7f85b12af694793950a560b471ad9de, f5be77523b4840c8a20a763a423b75c0; no voters: b176880e3fe5469a8b3393604567e4d2
I20250810 20:27:43.140003 32389 raft_consensus.cc:2802] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250810 20:27:43.140699 32389 raft_consensus.cc:491] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 1 FOLLOWER]: Starting leader election (detected failure of leader b176880e3fe5469a8b3393604567e4d2)
I20250810 20:27:43.141362 32389 raft_consensus.cc:3058] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 1 FOLLOWER]: Advancing to term 2
W20250810 20:27:43.151144 31953 leader_election.cc:336] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b176880e3fe5469a8b3393604567e4d2 (127.30.243.196:37121): Network error: Client connection negotiation failed: client connection to 127.30.243.196:37121: connect: Connection refused (error 111)
I20250810 20:27:43.164268 31886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22e4f3ebb6541d98c1c774c6b483731" candidate_uuid: "f5be77523b4840c8a20a763a423b75c0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "a7f85b12af694793950a560b471ad9de" is_pre_election: true
I20250810 20:27:43.171243 32389 raft_consensus.cc:513] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:43.173228 31886 raft_consensus.cc:2391] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f5be77523b4840c8a20a763a423b75c0 in current term 2: Already voted for candidate a7f85b12af694793950a560b471ad9de in this term.
I20250810 20:27:43.183362 31952 leader_election.cc:304] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5be77523b4840c8a20a763a423b75c0; no voters: a7f85b12af694793950a560b471ad9de, b176880e3fe5469a8b3393604567e4d2
I20250810 20:27:43.184700 32390 raft_consensus.cc:3058] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:27:43.187090 32389 leader_election.cc:290] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [CANDIDATE]: Term 2 election: Requested vote from peers b176880e3fe5469a8b3393604567e4d2 (127.30.243.196:37121), f5be77523b4840c8a20a763a423b75c0 (127.30.243.194:43945)
I20250810 20:27:43.192967 32019 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22e4f3ebb6541d98c1c774c6b483731" candidate_uuid: "a7f85b12af694793950a560b471ad9de" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f5be77523b4840c8a20a763a423b75c0"
I20250810 20:27:43.196923 32390 raft_consensus.cc:2747] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250810 20:27:43.201378 31820 leader_election.cc:336] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b176880e3fe5469a8b3393604567e4d2 (127.30.243.196:37121): Network error: Client connection negotiation failed: client connection to 127.30.243.196:37121: connect: Connection refused (error 111)
I20250810 20:27:43.206955 32019 raft_consensus.cc:2466] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a7f85b12af694793950a560b471ad9de in term 2.
I20250810 20:27:43.211792 31820 leader_election.cc:304] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a7f85b12af694793950a560b471ad9de, f5be77523b4840c8a20a763a423b75c0; no voters: b176880e3fe5469a8b3393604567e4d2
I20250810 20:27:43.213275 32389 raft_consensus.cc:2802] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 2 FOLLOWER]: Leader election won for term 2
I20250810 20:27:43.224390 32389 raft_consensus.cc:695] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 2 LEADER]: Becoming Leader. State: Replica: a7f85b12af694793950a560b471ad9de, State: Running, Role: LEADER
I20250810 20:27:43.225890 32389 consensus_queue.cc:237] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:43.642923 32019 raft_consensus.cc:1273] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 2 FOLLOWER]: Refusing update from remote peer a7f85b12af694793950a560b471ad9de: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250810 20:27:43.645471 32396 consensus_queue.cc:1035] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250810 20:27:43.685619 31820 consensus_peers.cc:489] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de -> Peer b176880e3fe5469a8b3393604567e4d2 (127.30.243.196:37121): Couldn't send request to peer b176880e3fe5469a8b3393604567e4d2. Status: Network error: Client connection negotiation failed: client connection to 127.30.243.196:37121: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250810 20:27:43.431169 32378 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 32372
W20250810 20:27:42.027546 32380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:43.802348 32372 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.776s user 0.535s sys 1.125s
W20250810 20:27:43.803161 32372 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.777s user 0.535s sys 1.126s
W20250810 20:27:43.804786 32382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:43.809237 32381 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1777 milliseconds
I20250810 20:27:43.809350 32372 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:27:43.810786 32372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:27:43.813591 32372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:27:43.815158 32372 hybrid_clock.cc:648] HybridClock initialized: now 1754857663815089 us; error 63 us; skew 500 ppm
I20250810 20:27:43.816265 32372 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:27:43.823103 32372 webserver.cc:489] Webserver started at http://127.30.243.254:34593/ using document root <none> and password file <none>
I20250810 20:27:43.824146 32372 fs_manager.cc:362] Metadata directory not provided
I20250810 20:27:43.824368 32372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:27:43.832608 32372 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20250810 20:27:43.837653 32406 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:43.838768 32372 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250810 20:27:43.839174 32372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492"
format_stamp: "Formatted at 2025-08-10 20:27:31 on dist-test-slave-7swh"
I20250810 20:27:43.841171 32372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:27:43.901103 32372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:27:43.902688 32372 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:27:43.903178 32372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:27:43.977779 32372 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:39215
I20250810 20:27:43.977910 32458 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:39215 every 8 connection(s)
I20250810 20:27:43.980706 32372 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 20:27:43.986706 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 32372
I20250810 20:27:43.987265 31695 kudu-admin-test.cc:735] Forcing unsafe config change on tserver f5be77523b4840c8a20a763a423b75c0
I20250810 20:27:43.991276 32459 sys_catalog.cc:263] Verifying existing consensus state
I20250810 20:27:43.997408 32459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492: Bootstrap starting.
I20250810 20:27:44.052121 32459 log.cc:826] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492: Log is configured to *not* fsync() on all Append() calls
I20250810 20:27:44.076568 32459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:27:44.077422 32459 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492: Bootstrap complete.
I20250810 20:27:44.099506 32459 raft_consensus.cc:357] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } }
I20250810 20:27:44.101634 32459 raft_consensus.cc:738] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fbacf430d3f4c22be9fbbe4fc1a3492, State: Initialized, Role: FOLLOWER
I20250810 20:27:44.102519 32459 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } }
I20250810 20:27:44.103084 32459 raft_consensus.cc:397] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:27:44.103377 32459 raft_consensus.cc:491] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:27:44.103705 32459 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:27:44.109997 32459 raft_consensus.cc:513] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } }
I20250810 20:27:44.110795 32459 leader_election.cc:304] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fbacf430d3f4c22be9fbbe4fc1a3492; no voters:
I20250810 20:27:44.113142 32459 leader_election.cc:290] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250810 20:27:44.113734 32463 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 20:27:44.117321 32463 raft_consensus.cc:695] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [term 2 LEADER]: Becoming Leader. State: Replica: 1fbacf430d3f4c22be9fbbe4fc1a3492, State: Running, Role: LEADER
I20250810 20:27:44.118444 32463 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } }
I20250810 20:27:44.119127 32459 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:27:44.130048 32464 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } } }
I20250810 20:27:44.130854 32464 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [sys.catalog]: This master's current role is: LEADER
I20250810 20:27:44.131856 32465 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fbacf430d3f4c22be9fbbe4fc1a3492. Latest consensus state: current_term: 2 leader_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fbacf430d3f4c22be9fbbe4fc1a3492" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 39215 } } }
I20250810 20:27:44.132841 32465 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492 [sys.catalog]: This master's current role is: LEADER
I20250810 20:27:44.138031 32470 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:27:44.158118 32470 catalog_manager.cc:671] Loaded metadata for table TestTable [id=d8ab97a7372f48d0bc8a05746d4e17c8]
I20250810 20:27:44.168386 32470 tablet_loader.cc:96] loaded metadata for tablet c22e4f3ebb6541d98c1c774c6b483731 (table TestTable [id=d8ab97a7372f48d0bc8a05746d4e17c8])
I20250810 20:27:44.170971 32470 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:27:44.176909 32470 catalog_manager.cc:1261] Loaded cluster ID: 5eca0178bcd2423ca9dbd3afdc54c736
I20250810 20:27:44.177407 32470 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:27:44.189838 32470 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:27:44.196422 32470 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 1fbacf430d3f4c22be9fbbe4fc1a3492: Loaded TSK: 0
I20250810 20:27:44.198565 32470 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250810 20:27:44.387362 32461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:44.388028 32461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:44.421654 32461 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250810 20:27:44.739758 31931 heartbeater.cc:344] Connected to a master server at 127.30.243.254:39215
I20250810 20:27:44.750061 32424 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "a7f85b12af694793950a560b471ad9de" instance_seqno: 1754857654625383) as {username='slave'} at 127.30.243.193:38473; Asking this server to re-register.
I20250810 20:27:44.750870 32064 heartbeater.cc:344] Connected to a master server at 127.30.243.254:39215
I20250810 20:27:44.755609 31931 heartbeater.cc:461] Registering TS with master...
I20250810 20:27:44.755286 32423 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" instance_seqno: 1754857656691314) as {username='slave'} at 127.30.243.194:50411; Asking this server to re-register.
I20250810 20:27:44.756348 31931 heartbeater.cc:507] Master 127.30.243.254:39215 requested a full tablet report, sending...
I20250810 20:27:44.757045 32064 heartbeater.cc:461] Registering TS with master...
I20250810 20:27:44.757879 32064 heartbeater.cc:507] Master 127.30.243.254:39215 requested a full tablet report, sending...
I20250810 20:27:44.760195 32423 ts_manager.cc:194] Registered new tserver with Master: a7f85b12af694793950a560b471ad9de (127.30.243.193:43391)
I20250810 20:27:44.761013 32424 ts_manager.cc:194] Registered new tserver with Master: f5be77523b4840c8a20a763a423b75c0 (127.30.243.194:43945)
I20250810 20:27:44.764882 32423 catalog_manager.cc:5582] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de reported cstate change: term changed from 1 to 2, leader changed from b176880e3fe5469a8b3393604567e4d2 (127.30.243.196) to a7f85b12af694793950a560b471ad9de (127.30.243.193). New cstate: current_term: 2 leader_uuid: "a7f85b12af694793950a560b471ad9de" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } health_report { overall_health: HEALTHY } } }
I20250810 20:27:44.868801 32197 heartbeater.cc:344] Connected to a master server at 127.30.243.254:39215
I20250810 20:27:44.871126 32424 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "724fa811237348079bf031f6b1e5057b" instance_seqno: 1754857658721412) as {username='slave'} at 127.30.243.195:33681; Asking this server to re-register.
I20250810 20:27:44.872685 32197 heartbeater.cc:461] Registering TS with master...
I20250810 20:27:44.873327 32197 heartbeater.cc:507] Master 127.30.243.254:39215 requested a full tablet report, sending...
I20250810 20:27:44.875676 32424 ts_manager.cc:194] Registered new tserver with Master: 724fa811237348079bf031f6b1e5057b (127.30.243.195:38463)
W20250810 20:27:45.869011 32488 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 32461
W20250810 20:27:45.887759 32488 kernel_stack_watchdog.cc:198] Thread 32461 stuck at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 20:27:46.124305 31820 consensus_peers.cc:489] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de -> Peer b176880e3fe5469a8b3393604567e4d2 (127.30.243.196:37121): Couldn't send request to peer b176880e3fe5469a8b3393604567e4d2. Status: Network error: Client connection negotiation failed: client connection to 127.30.243.196:37121: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250810 20:27:46.185674 32461 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.719s user 0.000s sys 0.002s
W20250810 20:27:46.328900 32461 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.862s user 0.009s sys 0.012s
I20250810 20:27:46.381271 32018 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "f5be77523b4840c8a20a763a423b75c0"
tablet_id: "c22e4f3ebb6541d98c1c774c6b483731"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "a7f85b12af694793950a560b471ad9de"
}
peers {
permanent_uuid: "f5be77523b4840c8a20a763a423b75c0"
}
}
from {username='slave'} at 127.0.0.1:53792
W20250810 20:27:46.382984 32018 raft_consensus.cc:2216] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } unsafe_config_change: true
I20250810 20:27:46.384243 32018 raft_consensus.cc:3058] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 2 FOLLOWER]: Advancing to term 3
I20250810 20:27:46.821264 32018 raft_consensus.cc:1238] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 3 FOLLOWER]: Rejecting Update request from peer a7f85b12af694793950a560b471ad9de for earlier term 2. Current term is 3. Ops: []
I20250810 20:27:46.822422 32389 consensus_queue.cc:1046] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250810 20:27:46.823733 32389 raft_consensus.cc:3053] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 2 LEADER]: Stepping down as leader of term 2
I20250810 20:27:46.823957 32389 raft_consensus.cc:738] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 2 LEADER]: Becoming Follower/Learner. State: Replica: a7f85b12af694793950a560b471ad9de, State: Running, Role: LEADER
I20250810 20:27:46.824551 32389 consensus_queue.cc:260] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } }
I20250810 20:27:46.825486 32389 raft_consensus.cc:3058] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 2 FOLLOWER]: Advancing to term 3
W20250810 20:27:47.098809 32488 debug-util.cc:398] Leaking SignalData structure 0x7b08000379e0 after lost signal to thread 32461
I20250810 20:27:47.952322 32517 raft_consensus.cc:491] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250810 20:27:47.952759 32517 raft_consensus.cc:513] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } unsafe_config_change: true
I20250810 20:27:47.953900 32517 leader_election.cc:290] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a7f85b12af694793950a560b471ad9de (127.30.243.193:43391)
I20250810 20:27:47.955050 31886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22e4f3ebb6541d98c1c774c6b483731" candidate_uuid: "f5be77523b4840c8a20a763a423b75c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a7f85b12af694793950a560b471ad9de" is_pre_election: true
I20250810 20:27:47.955503 31886 raft_consensus.cc:2466] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5be77523b4840c8a20a763a423b75c0 in term 3.
I20250810 20:27:47.956657 31952 leader_election.cc:304] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a7f85b12af694793950a560b471ad9de, f5be77523b4840c8a20a763a423b75c0; no voters:
I20250810 20:27:47.957329 32517 raft_consensus.cc:2802] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250810 20:27:47.957599 32517 raft_consensus.cc:491] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250810 20:27:47.957875 32517 raft_consensus.cc:3058] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 3 FOLLOWER]: Advancing to term 4
I20250810 20:27:47.962320 32517 raft_consensus.cc:513] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } unsafe_config_change: true
I20250810 20:27:47.963490 32517 leader_election.cc:290] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [CANDIDATE]: Term 4 election: Requested vote from peers a7f85b12af694793950a560b471ad9de (127.30.243.193:43391)
I20250810 20:27:47.964553 31886 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c22e4f3ebb6541d98c1c774c6b483731" candidate_uuid: "f5be77523b4840c8a20a763a423b75c0" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "a7f85b12af694793950a560b471ad9de"
I20250810 20:27:47.964993 31886 raft_consensus.cc:3058] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 3 FOLLOWER]: Advancing to term 4
I20250810 20:27:47.969890 31886 raft_consensus.cc:2466] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5be77523b4840c8a20a763a423b75c0 in term 4.
I20250810 20:27:47.970896 31952 leader_election.cc:304] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: a7f85b12af694793950a560b471ad9de, f5be77523b4840c8a20a763a423b75c0; no voters:
I20250810 20:27:47.971570 32517 raft_consensus.cc:2802] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 4 FOLLOWER]: Leader election won for term 4
I20250810 20:27:47.972494 32517 raft_consensus.cc:695] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 4 LEADER]: Becoming Leader. State: Replica: f5be77523b4840c8a20a763a423b75c0, State: Running, Role: LEADER
I20250810 20:27:47.973435 32517 consensus_queue.cc:237] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } unsafe_config_change: true
I20250810 20:27:47.982329 32424 catalog_manager.cc:5582] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 reported cstate change: term changed from 2 to 4, leader changed from a7f85b12af694793950a560b471ad9de (127.30.243.193) to f5be77523b4840c8a20a763a423b75c0 (127.30.243.194), now has a pending config: VOTER a7f85b12af694793950a560b471ad9de (127.30.243.193), VOTER f5be77523b4840c8a20a763a423b75c0 (127.30.243.194). New cstate: current_term: 4 leader_uuid: "f5be77523b4840c8a20a763a423b75c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b176880e3fe5469a8b3393604567e4d2" member_type: VOTER last_known_addr { host: "127.30.243.196" port: 37121 } } peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } unsafe_config_change: true }
I20250810 20:27:48.526121 31886 raft_consensus.cc:1273] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 4 FOLLOWER]: Refusing update from remote peer f5be77523b4840c8a20a763a423b75c0: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250810 20:27:48.527470 32524 consensus_queue.cc:1035] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250810 20:27:48.537231 32527 raft_consensus.cc:2953] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER b176880e3fe5469a8b3393604567e4d2 (127.30.243.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } unsafe_config_change: true }
I20250810 20:27:48.538260 31886 raft_consensus.cc:2953] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER b176880e3fe5469a8b3393604567e4d2 (127.30.243.196) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } unsafe_config_change: true }
I20250810 20:27:48.550046 32424 catalog_manager.cc:5582] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 reported cstate change: config changed from index -1 to 3, VOTER b176880e3fe5469a8b3393604567e4d2 (127.30.243.196) evicted, no longer has a pending config: VOTER a7f85b12af694793950a560b471ad9de (127.30.243.193), VOTER f5be77523b4840c8a20a763a423b75c0 (127.30.243.194). New cstate: current_term: 4 leader_uuid: "f5be77523b4840c8a20a763a423b75c0" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250810 20:27:48.561450 32424 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet c22e4f3ebb6541d98c1c774c6b483731 on TS b176880e3fe5469a8b3393604567e4d2: Not found: failed to reset TS proxy: Could not find TS for UUID b176880e3fe5469a8b3393604567e4d2
I20250810 20:27:48.579767 32018 consensus_queue.cc:237] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: true } } unsafe_config_change: true
I20250810 20:27:48.584481 31886 raft_consensus.cc:1273] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 4 FOLLOWER]: Refusing update from remote peer f5be77523b4840c8a20a763a423b75c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250810 20:27:48.586145 32524 consensus_queue.cc:1035] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250810 20:27:48.592541 32527 raft_consensus.cc:2953] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 724fa811237348079bf031f6b1e5057b (127.30.243.195) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: true } } unsafe_config_change: true }
I20250810 20:27:48.594210 31886 raft_consensus.cc:2953] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 724fa811237348079bf031f6b1e5057b (127.30.243.195) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: true } } unsafe_config_change: true }
I20250810 20:27:48.600373 32409 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c22e4f3ebb6541d98c1c774c6b483731 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250810 20:27:48.600914 31955 consensus_peers.cc:489] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 -> Peer 724fa811237348079bf031f6b1e5057b (127.30.243.195:38463): Couldn't send request to peer 724fa811237348079bf031f6b1e5057b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c22e4f3ebb6541d98c1c774c6b483731. This is attempt 1: this message will repeat every 5th retry.
I20250810 20:27:48.603375 32422 catalog_manager.cc:5582] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 reported cstate change: config changed from index 3 to 5, NON_VOTER 724fa811237348079bf031f6b1e5057b (127.30.243.195) added. New cstate: current_term: 4 leader_uuid: "f5be77523b4840c8a20a763a423b75c0" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250810 20:27:48.614342 32409 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet c22e4f3ebb6541d98c1c774c6b483731 on TS b176880e3fe5469a8b3393604567e4d2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b176880e3fe5469a8b3393604567e4d2
I20250810 20:27:49.167207 32539 ts_tablet_manager.cc:927] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: Initiating tablet copy from peer f5be77523b4840c8a20a763a423b75c0 (127.30.243.194:43945)
I20250810 20:27:49.170564 32539 tablet_copy_client.cc:323] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: tablet copy: Beginning tablet copy session from remote peer at address 127.30.243.194:43945
I20250810 20:27:49.182976 32039 tablet_copy_service.cc:140] P f5be77523b4840c8a20a763a423b75c0: Received BeginTabletCopySession request for tablet c22e4f3ebb6541d98c1c774c6b483731 from peer 724fa811237348079bf031f6b1e5057b ({username='slave'} at 127.30.243.195:39553)
I20250810 20:27:49.183471 32039 tablet_copy_service.cc:161] P f5be77523b4840c8a20a763a423b75c0: Beginning new tablet copy session on tablet c22e4f3ebb6541d98c1c774c6b483731 from peer 724fa811237348079bf031f6b1e5057b at {username='slave'} at 127.30.243.195:39553: session id = 724fa811237348079bf031f6b1e5057b-c22e4f3ebb6541d98c1c774c6b483731
I20250810 20:27:49.189124 32039 tablet_copy_source_session.cc:215] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0: Tablet Copy: opened 0 blocks and 1 log segments
I20250810 20:27:49.195550 32539 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c22e4f3ebb6541d98c1c774c6b483731. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:27:49.216930 32539 tablet_copy_client.cc:806] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: tablet copy: Starting download of 0 data blocks...
I20250810 20:27:49.217572 32539 tablet_copy_client.cc:670] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: tablet copy: Starting download of 1 WAL segments...
I20250810 20:27:49.221240 32539 tablet_copy_client.cc:538] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250810 20:27:49.227842 32539 tablet_bootstrap.cc:492] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: Bootstrap starting.
I20250810 20:27:49.240525 32539 log.cc:826] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: Log is configured to *not* fsync() on all Append() calls
I20250810 20:27:49.253258 32539 tablet_bootstrap.cc:492] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:27:49.254083 32539 tablet_bootstrap.cc:492] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: Bootstrap complete.
I20250810 20:27:49.254719 32539 ts_tablet_manager.cc:1397] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: Time spent bootstrapping tablet: real 0.027s user 0.018s sys 0.005s
I20250810 20:27:49.273097 32539 raft_consensus.cc:357] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: true } } unsafe_config_change: true
I20250810 20:27:49.274045 32539 raft_consensus.cc:738] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 724fa811237348079bf031f6b1e5057b, State: Initialized, Role: LEARNER
I20250810 20:27:49.274760 32539 consensus_queue.cc:260] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: true } } unsafe_config_change: true
I20250810 20:27:49.278396 32539 ts_tablet_manager.cc:1428] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b: Time spent starting tablet: real 0.023s user 0.022s sys 0.000s
I20250810 20:27:49.280272 32039 tablet_copy_service.cc:342] P f5be77523b4840c8a20a763a423b75c0: Request end of tablet copy session 724fa811237348079bf031f6b1e5057b-c22e4f3ebb6541d98c1c774c6b483731 received from {username='slave'} at 127.30.243.195:39553
I20250810 20:27:49.280658 32039 tablet_copy_service.cc:434] P f5be77523b4840c8a20a763a423b75c0: ending tablet copy session 724fa811237348079bf031f6b1e5057b-c22e4f3ebb6541d98c1c774c6b483731 on tablet c22e4f3ebb6541d98c1c774c6b483731 with peer 724fa811237348079bf031f6b1e5057b
I20250810 20:27:49.715049 32152 raft_consensus.cc:1215] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250810 20:27:49.745543 32060 debug-util.cc:398] Leaking SignalData structure 0x7b08000c7160 after lost signal to thread 31936
W20250810 20:27:49.943396 32409 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet c22e4f3ebb6541d98c1c774c6b483731 on TS b176880e3fe5469a8b3393604567e4d2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID b176880e3fe5469a8b3393604567e4d2
I20250810 20:27:50.235714 32546 raft_consensus.cc:1062] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0: attempting to promote NON_VOTER 724fa811237348079bf031f6b1e5057b to VOTER
I20250810 20:27:50.237650 32546 consensus_queue.cc:237] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: false } } unsafe_config_change: true
I20250810 20:27:50.243909 32152 raft_consensus.cc:1273] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b [term 4 LEARNER]: Refusing update from remote peer f5be77523b4840c8a20a763a423b75c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250810 20:27:50.244606 31886 raft_consensus.cc:1273] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 4 FOLLOWER]: Refusing update from remote peer f5be77523b4840c8a20a763a423b75c0: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250810 20:27:50.245831 32549 consensus_queue.cc:1035] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250810 20:27:50.246685 32546 consensus_queue.cc:1035] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250810 20:27:50.259793 32549 raft_consensus.cc:2953] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 724fa811237348079bf031f6b1e5057b (127.30.243.195) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: false } } unsafe_config_change: true }
I20250810 20:27:50.263585 32152 raft_consensus.cc:2953] T c22e4f3ebb6541d98c1c774c6b483731 P 724fa811237348079bf031f6b1e5057b [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 724fa811237348079bf031f6b1e5057b (127.30.243.195) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: false } } unsafe_config_change: true }
I20250810 20:27:50.267601 31886 raft_consensus.cc:2953] T c22e4f3ebb6541d98c1c774c6b483731 P a7f85b12af694793950a560b471ad9de [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 724fa811237348079bf031f6b1e5057b (127.30.243.195) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: false } } unsafe_config_change: true }
I20250810 20:27:50.272007 32422 catalog_manager.cc:5582] T c22e4f3ebb6541d98c1c774c6b483731 P f5be77523b4840c8a20a763a423b75c0 reported cstate change: config changed from index 5 to 6, 724fa811237348079bf031f6b1e5057b (127.30.243.195) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "f5be77523b4840c8a20a763a423b75c0" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "a7f85b12af694793950a560b471ad9de" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5be77523b4840c8a20a763a423b75c0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "724fa811237348079bf031f6b1e5057b" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 38463 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250810 20:27:50.391073 31695 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250810 20:27:50.408537 31695 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "c22e4f3ebb6541d98c1c774c6b483731"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "a7f85b12af694793950a560b471ad9de"
rpc_addresses {
host: "127.30.243.193"
port: 43391
}
}
ts_infos {
permanent_uuid: "f5be77523b4840c8a20a763a423b75c0"
rpc_addresses {
host: "127.30.243.194"
port: 43945
}
}
ts_infos {
permanent_uuid: "724fa811237348079bf031f6b1e5057b"
rpc_addresses {
host: "127.30.243.195"
port: 38463
}
}
I20250810 20:27:50.411914 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 31802
I20250810 20:27:50.442510 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 31935
I20250810 20:27:50.475914 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 32068
I20250810 20:27:50.505605 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 32372
2025-08-10T20:27:50Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (21652 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250810 20:27:50.580582 31695 test_util.cc:276] Using random seed: -1542055612
I20250810 20:27:50.587670 31695 ts_itest-base.cc:115] Starting cluster with:
I20250810 20:27:50.587885 31695 ts_itest-base.cc:116] --------------
I20250810 20:27:50.588091 31695 ts_itest-base.cc:117] 3 tablet servers
I20250810 20:27:50.588235 31695 ts_itest-base.cc:118] 3 replicas per TS
I20250810 20:27:50.588379 31695 ts_itest-base.cc:119] --------------
2025-08-10T20:27:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T20:27:50Z Disabled control of system clock
I20250810 20:27:50.628096 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:45613
--webserver_interface=127.30.243.254
--webserver_port=0
--builtin_ntp_servers=127.30.243.212:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:45613
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250810 20:27:50.948678 32567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:50.949354 32567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:27:50.949837 32567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:50.990461 32567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:27:50.990868 32567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:27:50.991189 32567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:27:50.991436 32567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:27:51.029032 32567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:45613
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:45613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:27:51.030486 32567 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:27:51.032305 32567 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:27:51.045110 32573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:52.447794 32572 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 32567
W20250810 20:27:52.924516 32572 kernel_stack_watchdog.cc:198] Thread 32567 stuck at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 20:27:51.045396 32574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:52.925855 32567 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.880s user 0.659s sys 1.220s
W20250810 20:27:52.926335 32567 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.881s user 0.659s sys 1.220s
W20250810 20:27:52.928313 32576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:52.931026 32575 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1879 milliseconds
I20250810 20:27:52.931140 32567 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:27:52.932433 32567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:27:52.935384 32567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:27:52.936932 32567 hybrid_clock.cc:648] HybridClock initialized: now 1754857672936866 us; error 70 us; skew 500 ppm
I20250810 20:27:52.937808 32567 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:27:52.945536 32567 webserver.cc:489] Webserver started at http://127.30.243.254:43689/ using document root <none> and password file <none>
I20250810 20:27:52.946681 32567 fs_manager.cc:362] Metadata directory not provided
I20250810 20:27:52.946985 32567 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:27:52.947475 32567 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:27:52.952376 32567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "30c9097ee4944a279b88dd8099bb5be2"
format_stamp: "Formatted at 2025-08-10 20:27:52 on dist-test-slave-7swh"
I20250810 20:27:52.953617 32567 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "30c9097ee4944a279b88dd8099bb5be2"
format_stamp: "Formatted at 2025-08-10 20:27:52 on dist-test-slave-7swh"
I20250810 20:27:52.961640 32567 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250810 20:27:52.970597 32583 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:52.972118 32567 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20250810 20:27:52.972519 32567 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "30c9097ee4944a279b88dd8099bb5be2"
format_stamp: "Formatted at 2025-08-10 20:27:52 on dist-test-slave-7swh"
I20250810 20:27:52.972914 32567 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:27:53.037479 32567 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:27:53.039088 32567 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:27:53.039566 32567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:27:53.122136 32567 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:45613
I20250810 20:27:53.122210 32634 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:45613 every 8 connection(s)
I20250810 20:27:53.125038 32567 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 20:27:53.130834 32635 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:27:53.131184 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 32567
I20250810 20:27:53.131603 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 20:27:53.159447 32635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2: Bootstrap starting.
I20250810 20:27:53.166566 32635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2: Neither blocks nor log segments found. Creating new log.
I20250810 20:27:53.169066 32635 log.cc:826] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2: Log is configured to *not* fsync() on all Append() calls
I20250810 20:27:53.176872 32635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2: No bootstrap required, opened a new log
I20250810 20:27:53.197316 32635 raft_consensus.cc:357] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c9097ee4944a279b88dd8099bb5be2" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 45613 } }
I20250810 20:27:53.198071 32635 raft_consensus.cc:383] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:27:53.198276 32635 raft_consensus.cc:738] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30c9097ee4944a279b88dd8099bb5be2, State: Initialized, Role: FOLLOWER
I20250810 20:27:53.198989 32635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c9097ee4944a279b88dd8099bb5be2" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 45613 } }
I20250810 20:27:53.199549 32635 raft_consensus.cc:397] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:27:53.199797 32635 raft_consensus.cc:491] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:27:53.200153 32635 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:27:53.223676 32635 raft_consensus.cc:513] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c9097ee4944a279b88dd8099bb5be2" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 45613 } }
I20250810 20:27:53.224586 32635 leader_election.cc:304] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30c9097ee4944a279b88dd8099bb5be2; no voters:
I20250810 20:27:53.226459 32635 leader_election.cc:290] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:27:53.227382 32640 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:27:53.229916 32640 raft_consensus.cc:695] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [term 1 LEADER]: Becoming Leader. State: Replica: 30c9097ee4944a279b88dd8099bb5be2, State: Running, Role: LEADER
I20250810 20:27:53.230717 32640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c9097ee4944a279b88dd8099bb5be2" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 45613 } }
I20250810 20:27:53.232240 32635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:27:53.241391 32642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30c9097ee4944a279b88dd8099bb5be2. Latest consensus state: current_term: 1 leader_uuid: "30c9097ee4944a279b88dd8099bb5be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c9097ee4944a279b88dd8099bb5be2" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 45613 } } }
I20250810 20:27:53.241861 32641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30c9097ee4944a279b88dd8099bb5be2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30c9097ee4944a279b88dd8099bb5be2" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 45613 } } }
I20250810 20:27:53.242565 32641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [sys.catalog]: This master's current role is: LEADER
I20250810 20:27:53.242554 32642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2 [sys.catalog]: This master's current role is: LEADER
I20250810 20:27:53.249081 32649 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:27:53.265151 32649 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:27:53.284778 32649 catalog_manager.cc:1349] Generated new cluster ID: d459e1dcf7e843c593ff73834b5a54c2
I20250810 20:27:53.285193 32649 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:27:53.318702 32649 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 20:27:53.320484 32649 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:27:53.342344 32649 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 30c9097ee4944a279b88dd8099bb5be2: Generated new TSK 0
I20250810 20:27:53.343629 32649 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 20:27:53.369161 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:0
--local_ip_for_outbound_sockets=127.30.243.193
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:45613
--builtin_ntp_servers=127.30.243.212:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250810 20:27:53.709893 32659 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250810 20:27:53.710721 32659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:53.711024 32659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:27:53.711519 32659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:53.745060 32659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:27:53.746016 32659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:27:53.786669 32659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:45613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:27:53.788159 32659 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:27:53.790208 32659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:27:53.805119 32665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:53.807104 32666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:55.128841 32668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:55.132505 32667 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1320 milliseconds
I20250810 20:27:55.132660 32659 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:27:55.134606 32659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:27:55.139535 32659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:27:55.141144 32659 hybrid_clock.cc:648] HybridClock initialized: now 1754857675141068 us; error 92 us; skew 500 ppm
I20250810 20:27:55.142032 32659 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:27:55.149587 32659 webserver.cc:489] Webserver started at http://127.30.243.193:43651/ using document root <none> and password file <none>
I20250810 20:27:55.150597 32659 fs_manager.cc:362] Metadata directory not provided
I20250810 20:27:55.150838 32659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:27:55.151362 32659 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:27:55.156649 32659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5c7aad16bb304fa5903e6036836dd27d"
format_stamp: "Formatted at 2025-08-10 20:27:55 on dist-test-slave-7swh"
I20250810 20:27:55.157904 32659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5c7aad16bb304fa5903e6036836dd27d"
format_stamp: "Formatted at 2025-08-10 20:27:55 on dist-test-slave-7swh"
I20250810 20:27:55.166111 32659 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.010s sys 0.000s
I20250810 20:27:55.173009 32675 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:55.174289 32659 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20250810 20:27:55.174662 32659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5c7aad16bb304fa5903e6036836dd27d"
format_stamp: "Formatted at 2025-08-10 20:27:55 on dist-test-slave-7swh"
I20250810 20:27:55.175088 32659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:27:55.227061 32659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:27:55.228614 32659 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:27:55.229116 32659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:27:55.232277 32659 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:27:55.237344 32659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:27:55.237563 32659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:55.237846 32659 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:27:55.238005 32659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:55.416381 32659 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:45459
I20250810 20:27:55.416556 319 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:45459 every 8 connection(s)
I20250810 20:27:55.419117 32659 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 20:27:55.421860 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 32659
I20250810 20:27:55.422418 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 20:27:55.433777 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.194:0
--local_ip_for_outbound_sockets=127.30.243.194
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:45613
--builtin_ntp_servers=127.30.243.212:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250810 20:27:55.456760 320 heartbeater.cc:344] Connected to a master server at 127.30.243.254:45613
I20250810 20:27:55.457535 320 heartbeater.cc:461] Registering TS with master...
I20250810 20:27:55.459125 320 heartbeater.cc:507] Master 127.30.243.254:45613 requested a full tablet report, sending...
I20250810 20:27:55.463135 32600 ts_manager.cc:194] Registered new tserver with Master: 5c7aad16bb304fa5903e6036836dd27d (127.30.243.193:45459)
I20250810 20:27:55.466027 32600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:38609
W20250810 20:27:55.763233 324 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250810 20:27:55.763880 324 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:55.764155 324 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:27:55.764642 324 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:55.796715 324 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:27:55.797586 324 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.194
I20250810 20:27:55.836071 324 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:45613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.194
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:27:55.837507 324 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:27:55.839272 324 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:27:55.853410 330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:27:56.471251 320 heartbeater.cc:499] Master 127.30.243.254:45613 was elected leader, sending a full tablet report...
W20250810 20:27:55.855547 331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:55.857271 333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:57.074805 332 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1220 milliseconds
I20250810 20:27:57.074869 324 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:27:57.081490 324 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:27:57.084559 324 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:27:57.086079 324 hybrid_clock.cc:648] HybridClock initialized: now 1754857677085978 us; error 120 us; skew 500 ppm
I20250810 20:27:57.086918 324 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:27:57.094228 324 webserver.cc:489] Webserver started at http://127.30.243.194:37545/ using document root <none> and password file <none>
I20250810 20:27:57.095250 324 fs_manager.cc:362] Metadata directory not provided
I20250810 20:27:57.095454 324 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:27:57.095911 324 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:27:57.100426 324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "889a9293101b4417b0e8d6c2105b0ce0"
format_stamp: "Formatted at 2025-08-10 20:27:57 on dist-test-slave-7swh"
I20250810 20:27:57.101526 324 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "889a9293101b4417b0e8d6c2105b0ce0"
format_stamp: "Formatted at 2025-08-10 20:27:57 on dist-test-slave-7swh"
I20250810 20:27:57.109131 324 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.008s
I20250810 20:27:57.115541 340 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:57.116750 324 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 20:27:57.117085 324 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "889a9293101b4417b0e8d6c2105b0ce0"
format_stamp: "Formatted at 2025-08-10 20:27:57 on dist-test-slave-7swh"
I20250810 20:27:57.117414 324 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:27:57.204566 324 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:27:57.206084 324 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:27:57.206501 324 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:27:57.209152 324 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:27:57.213507 324 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:27:57.213732 324 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:57.213935 324 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:27:57.214084 324 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:57.354064 324 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.194:34163
I20250810 20:27:57.354362 453 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.194:34163 every 8 connection(s)
I20250810 20:27:57.356895 324 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 20:27:57.360133 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 324
I20250810 20:27:57.360738 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 20:27:57.369315 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.195:0
--local_ip_for_outbound_sockets=127.30.243.195
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:45613
--builtin_ntp_servers=127.30.243.212:37463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250810 20:27:57.384104 454 heartbeater.cc:344] Connected to a master server at 127.30.243.254:45613
I20250810 20:27:57.384558 454 heartbeater.cc:461] Registering TS with master...
I20250810 20:27:57.385653 454 heartbeater.cc:507] Master 127.30.243.254:45613 requested a full tablet report, sending...
I20250810 20:27:57.388200 32600 ts_manager.cc:194] Registered new tserver with Master: 889a9293101b4417b0e8d6c2105b0ce0 (127.30.243.194:34163)
I20250810 20:27:57.390256 32600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.194:49557
W20250810 20:27:57.684209 458 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250810 20:27:57.684810 458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:27:57.685043 458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:27:57.685470 458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:27:57.717602 458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:27:57.718578 458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.195
I20250810 20:27:57.754961 458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:37463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:45613
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.195
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:27:57.756253 458 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:27:57.757810 458 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:27:57.769577 464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:27:58.394217 454 heartbeater.cc:499] Master 127.30.243.254:45613 was elected leader, sending a full tablet report...
W20250810 20:27:57.771500 465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:58.997752 467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:27:58.999293 466 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1224 milliseconds
I20250810 20:27:58.999434 458 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:27:59.000633 458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:27:59.003602 458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:27:59.005075 458 hybrid_clock.cc:648] HybridClock initialized: now 1754857679005015 us; error 76 us; skew 500 ppm
I20250810 20:27:59.005872 458 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:27:59.013116 458 webserver.cc:489] Webserver started at http://127.30.243.195:35311/ using document root <none> and password file <none>
I20250810 20:27:59.014153 458 fs_manager.cc:362] Metadata directory not provided
I20250810 20:27:59.014374 458 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:27:59.015236 458 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:27:59.020614 458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "c41a7e351fd44aa6bc06c802faf38da5"
format_stamp: "Formatted at 2025-08-10 20:27:59 on dist-test-slave-7swh"
I20250810 20:27:59.021912 458 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "c41a7e351fd44aa6bc06c802faf38da5"
format_stamp: "Formatted at 2025-08-10 20:27:59 on dist-test-slave-7swh"
I20250810 20:27:59.029984 458 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250810 20:27:59.037822 474 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:59.039085 458 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.002s
I20250810 20:27:59.039436 458 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "c41a7e351fd44aa6bc06c802faf38da5"
format_stamp: "Formatted at 2025-08-10 20:27:59 on dist-test-slave-7swh"
I20250810 20:27:59.039803 458 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:27:59.095016 458 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:27:59.096479 458 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:27:59.096922 458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:27:59.099510 458 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:27:59.104054 458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:27:59.104368 458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:59.104604 458 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:27:59.104763 458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:27:59.254168 458 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.195:43857
I20250810 20:27:59.254273 587 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.195:43857 every 8 connection(s)
I20250810 20:27:59.256809 458 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 20:27:59.259040 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 458
I20250810 20:27:59.259748 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 20:27:59.284147 588 heartbeater.cc:344] Connected to a master server at 127.30.243.254:45613
I20250810 20:27:59.284631 588 heartbeater.cc:461] Registering TS with master...
I20250810 20:27:59.285723 588 heartbeater.cc:507] Master 127.30.243.254:45613 requested a full tablet report, sending...
I20250810 20:27:59.288336 32600 ts_manager.cc:194] Registered new tserver with Master: c41a7e351fd44aa6bc06c802faf38da5 (127.30.243.195:43857)
I20250810 20:27:59.289603 32600 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.195:42671
I20250810 20:27:59.296135 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:27:59.334990 32600 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33100:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250810 20:27:59.354539 32600 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 20:27:59.420533 388 tablet_service.cc:1468] Processing CreateTablet for tablet 96e14484a3eb458b8e1e24647555ed30 (DEFAULT_TABLE table=TestTable [id=0023c4e56d63431aa058308aa9abef0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:27:59.424340 522 tablet_service.cc:1468] Processing CreateTablet for tablet 96e14484a3eb458b8e1e24647555ed30 (DEFAULT_TABLE table=TestTable [id=0023c4e56d63431aa058308aa9abef0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:27:59.425834 388 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e14484a3eb458b8e1e24647555ed30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:27:59.426267 522 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e14484a3eb458b8e1e24647555ed30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:27:59.428409 32723 tablet_service.cc:1468] Processing CreateTablet for tablet 96e14484a3eb458b8e1e24647555ed30 (DEFAULT_TABLE table=TestTable [id=0023c4e56d63431aa058308aa9abef0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:27:59.430471 32723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96e14484a3eb458b8e1e24647555ed30. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:27:59.456933 607 tablet_bootstrap.cc:492] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5: Bootstrap starting.
I20250810 20:27:59.458050 608 tablet_bootstrap.cc:492] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0: Bootstrap starting.
I20250810 20:27:59.464640 609 tablet_bootstrap.cc:492] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d: Bootstrap starting.
I20250810 20:27:59.465265 607 tablet_bootstrap.cc:654] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5: Neither blocks nor log segments found. Creating new log.
I20250810 20:27:59.466687 608 tablet_bootstrap.cc:654] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0: Neither blocks nor log segments found. Creating new log.
I20250810 20:27:59.467830 607 log.cc:826] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5: Log is configured to *not* fsync() on all Append() calls
I20250810 20:27:59.469187 608 log.cc:826] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0: Log is configured to *not* fsync() on all Append() calls
I20250810 20:27:59.473865 609 tablet_bootstrap.cc:654] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d: Neither blocks nor log segments found. Creating new log.
I20250810 20:27:59.474815 607 tablet_bootstrap.cc:492] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5: No bootstrap required, opened a new log
I20250810 20:27:59.475459 607 ts_tablet_manager.cc:1397] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5: Time spent bootstrapping tablet: real 0.019s user 0.016s sys 0.002s
I20250810 20:27:59.476496 609 log.cc:826] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d: Log is configured to *not* fsync() on all Append() calls
I20250810 20:27:59.477408 608 tablet_bootstrap.cc:492] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0: No bootstrap required, opened a new log
I20250810 20:27:59.477843 608 ts_tablet_manager.cc:1397] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.013s
I20250810 20:27:59.483176 609 tablet_bootstrap.cc:492] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d: No bootstrap required, opened a new log
I20250810 20:27:59.484187 609 ts_tablet_manager.cc:1397] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d: Time spent bootstrapping tablet: real 0.020s user 0.014s sys 0.004s
I20250810 20:27:59.497279 607 raft_consensus.cc:357] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:27:59.498361 607 raft_consensus.cc:738] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c41a7e351fd44aa6bc06c802faf38da5, State: Initialized, Role: FOLLOWER
I20250810 20:27:59.499172 607 consensus_queue.cc:260] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:27:59.502718 588 heartbeater.cc:499] Master 127.30.243.254:45613 was elected leader, sending a full tablet report...
I20250810 20:27:59.505010 607 ts_tablet_manager.cc:1428] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5: Time spent starting tablet: real 0.029s user 0.018s sys 0.009s
I20250810 20:27:59.505892 608 raft_consensus.cc:357] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:27:59.506860 608 raft_consensus.cc:738] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 889a9293101b4417b0e8d6c2105b0ce0, State: Initialized, Role: FOLLOWER
I20250810 20:27:59.507654 608 consensus_queue.cc:260] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:27:59.512511 608 ts_tablet_manager.cc:1428] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0: Time spent starting tablet: real 0.034s user 0.022s sys 0.011s
W20250810 20:27:59.512531 589 tablet.cc:2378] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:27:59.513330 609 raft_consensus.cc:357] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:27:59.514271 609 raft_consensus.cc:738] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c7aad16bb304fa5903e6036836dd27d, State: Initialized, Role: FOLLOWER
I20250810 20:27:59.515074 609 consensus_queue.cc:260] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:27:59.519100 609 ts_tablet_manager.cc:1428] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d: Time spent starting tablet: real 0.035s user 0.026s sys 0.009s
I20250810 20:27:59.536760 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:27:59.540689 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5c7aad16bb304fa5903e6036836dd27d to finish bootstrapping
I20250810 20:27:59.554311 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 889a9293101b4417b0e8d6c2105b0ce0 to finish bootstrapping
I20250810 20:27:59.567165 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c41a7e351fd44aa6bc06c802faf38da5 to finish bootstrapping
I20250810 20:27:59.613202 32743 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "96e14484a3eb458b8e1e24647555ed30"
dest_uuid: "5c7aad16bb304fa5903e6036836dd27d"
from {username='slave'} at 127.0.0.1:47794
I20250810 20:27:59.613775 32743 raft_consensus.cc:491] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250810 20:27:59.614055 32743 raft_consensus.cc:3058] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 0 FOLLOWER]: Advancing to term 1
W20250810 20:27:59.614988 455 tablet.cc:2378] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:27:59.619155 32743 raft_consensus.cc:513] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:27:59.621381 32743 leader_election.cc:290] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [CANDIDATE]: Term 1 election: Requested vote from peers c41a7e351fd44aa6bc06c802faf38da5 (127.30.243.195:43857), 889a9293101b4417b0e8d6c2105b0ce0 (127.30.243.194:34163)
I20250810 20:27:59.630384 31695 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250810 20:27:59.635983 543 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96e14484a3eb458b8e1e24647555ed30" candidate_uuid: "5c7aad16bb304fa5903e6036836dd27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c41a7e351fd44aa6bc06c802faf38da5"
I20250810 20:27:59.635986 408 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96e14484a3eb458b8e1e24647555ed30" candidate_uuid: "5c7aad16bb304fa5903e6036836dd27d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "889a9293101b4417b0e8d6c2105b0ce0"
I20250810 20:27:59.636787 543 raft_consensus.cc:3058] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:27:59.636787 408 raft_consensus.cc:3058] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:27:59.644009 543 raft_consensus.cc:2466] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c7aad16bb304fa5903e6036836dd27d in term 1.
I20250810 20:27:59.644009 408 raft_consensus.cc:2466] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c7aad16bb304fa5903e6036836dd27d in term 1.
I20250810 20:27:59.645334 32678 leader_election.cc:304] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5c7aad16bb304fa5903e6036836dd27d, 889a9293101b4417b0e8d6c2105b0ce0; no voters:
I20250810 20:27:59.646113 615 raft_consensus.cc:2802] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:27:59.647696 615 raft_consensus.cc:695] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 1 LEADER]: Becoming Leader. State: Replica: 5c7aad16bb304fa5903e6036836dd27d, State: Running, Role: LEADER
I20250810 20:27:59.648551 615 consensus_queue.cc:237] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:27:59.659945 32598 catalog_manager.cc:5582] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c7aad16bb304fa5903e6036836dd27d (127.30.243.193). New cstate: current_term: 1 leader_uuid: "5c7aad16bb304fa5903e6036836dd27d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } health_report { overall_health: HEALTHY } } }
W20250810 20:27:59.702885 321 tablet.cc:2378] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:27:59.736640 31695 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250810 20:27:59.942668 31695 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250810 20:28:00.053051 615 consensus_queue.cc:1035] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [LEADER]: Connected to new peer: Peer: permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 20:28:00.069482 620 consensus_queue.cc:1035] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [LEADER]: Connected to new peer: Peer: permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 20:28:01.972507 32743 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "96e14484a3eb458b8e1e24647555ed30"
dest_uuid: "5c7aad16bb304fa5903e6036836dd27d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47798
I20250810 20:28:01.973055 32743 raft_consensus.cc:604] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 1 LEADER]: Received request to transfer leadership
I20250810 20:28:02.144431 650 raft_consensus.cc:991] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d: : Instructing follower 889a9293101b4417b0e8d6c2105b0ce0 to start an election
I20250810 20:28:02.144807 634 raft_consensus.cc:1079] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 1 LEADER]: Signalling peer 889a9293101b4417b0e8d6c2105b0ce0 to start an election
I20250810 20:28:02.146327 408 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "96e14484a3eb458b8e1e24647555ed30"
dest_uuid: "889a9293101b4417b0e8d6c2105b0ce0"
from {username='slave'} at 127.30.243.193:57821
I20250810 20:28:02.146860 408 raft_consensus.cc:491] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250810 20:28:02.147150 408 raft_consensus.cc:3058] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:28:02.151546 408 raft_consensus.cc:513] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:28:02.153713 408 leader_election.cc:290] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [CANDIDATE]: Term 2 election: Requested vote from peers c41a7e351fd44aa6bc06c802faf38da5 (127.30.243.195:43857), 5c7aad16bb304fa5903e6036836dd27d (127.30.243.193:45459)
I20250810 20:28:02.164809 543 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96e14484a3eb458b8e1e24647555ed30" candidate_uuid: "889a9293101b4417b0e8d6c2105b0ce0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "c41a7e351fd44aa6bc06c802faf38da5"
I20250810 20:28:02.165400 543 raft_consensus.cc:3058] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5 [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:28:02.166590 32743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96e14484a3eb458b8e1e24647555ed30" candidate_uuid: "889a9293101b4417b0e8d6c2105b0ce0" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "5c7aad16bb304fa5903e6036836dd27d"
I20250810 20:28:02.167266 32743 raft_consensus.cc:3053] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 1 LEADER]: Stepping down as leader of term 1
I20250810 20:28:02.167586 32743 raft_consensus.cc:738] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5c7aad16bb304fa5903e6036836dd27d, State: Running, Role: LEADER
I20250810 20:28:02.168244 32743 consensus_queue.cc:260] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:28:02.169718 32743 raft_consensus.cc:3058] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:28:02.171743 543 raft_consensus.cc:2466] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889a9293101b4417b0e8d6c2105b0ce0 in term 2.
I20250810 20:28:02.172927 344 leader_election.cc:304] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 889a9293101b4417b0e8d6c2105b0ce0, c41a7e351fd44aa6bc06c802faf38da5; no voters:
I20250810 20:28:02.174867 32743 raft_consensus.cc:2466] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 889a9293101b4417b0e8d6c2105b0ce0 in term 2.
I20250810 20:28:02.175232 654 raft_consensus.cc:2802] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 20:28:02.177143 654 raft_consensus.cc:695] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [term 2 LEADER]: Becoming Leader. State: Replica: 889a9293101b4417b0e8d6c2105b0ce0, State: Running, Role: LEADER
I20250810 20:28:02.177968 654 consensus_queue.cc:237] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } }
I20250810 20:28:02.185747 32597 catalog_manager.cc:5582] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 reported cstate change: term changed from 1 to 2, leader changed from 5c7aad16bb304fa5903e6036836dd27d (127.30.243.193) to 889a9293101b4417b0e8d6c2105b0ce0 (127.30.243.194). New cstate: current_term: 2 leader_uuid: "889a9293101b4417b0e8d6c2105b0ce0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "889a9293101b4417b0e8d6c2105b0ce0" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 34163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 } health_report { overall_health: UNKNOWN } } }
I20250810 20:28:02.754683 32743 raft_consensus.cc:1273] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 2 FOLLOWER]: Refusing update from remote peer 889a9293101b4417b0e8d6c2105b0ce0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250810 20:28:02.756080 661 consensus_queue.cc:1035] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c7aad16bb304fa5903e6036836dd27d" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 45459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250810 20:28:02.766570 543 raft_consensus.cc:1273] T 96e14484a3eb458b8e1e24647555ed30 P c41a7e351fd44aa6bc06c802faf38da5 [term 2 FOLLOWER]: Refusing update from remote peer 889a9293101b4417b0e8d6c2105b0ce0: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250810 20:28:02.769204 666 consensus_queue.cc:1035] T 96e14484a3eb458b8e1e24647555ed30 P 889a9293101b4417b0e8d6c2105b0ce0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c41a7e351fd44aa6bc06c802faf38da5" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250810 20:28:04.750990 32743 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "96e14484a3eb458b8e1e24647555ed30"
dest_uuid: "5c7aad16bb304fa5903e6036836dd27d"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47806
I20250810 20:28:04.751607 32743 raft_consensus.cc:604] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 2 FOLLOWER]: Received request to transfer leadership
I20250810 20:28:04.751933 32743 raft_consensus.cc:612] T 96e14484a3eb458b8e1e24647555ed30 P 5c7aad16bb304fa5903e6036836dd27d [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250810 20:28:05.792513 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 32659
I20250810 20:28:05.819782 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 324
I20250810 20:28:05.844962 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 458
I20250810 20:28:05.871407 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 32567
2025-08-10T20:28:05Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (15347 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250810 20:28:05.927042 31695 test_util.cc:276] Using random seed: -1526709146
I20250810 20:28:05.931787 31695 ts_itest-base.cc:115] Starting cluster with:
I20250810 20:28:05.931967 31695 ts_itest-base.cc:116] --------------
I20250810 20:28:05.932091 31695 ts_itest-base.cc:117] 3 tablet servers
I20250810 20:28:05.932205 31695 ts_itest-base.cc:118] 3 replicas per TS
I20250810 20:28:05.932329 31695 ts_itest-base.cc:119] --------------
2025-08-10T20:28:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T20:28:05Z Disabled control of system clock
I20250810 20:28:05.972543 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:41319
--webserver_interface=127.30.243.254
--webserver_port=0
--builtin_ntp_servers=127.30.243.212:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:41319 with env {}
W20250810 20:28:06.312091 700 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:06.312722 700 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:06.313205 700 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:06.345608 700 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:28:06.345937 700 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:06.346254 700 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:28:06.346480 700 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:28:06.384542 700 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:41319
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:41319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:06.385984 700 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:06.387822 700 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:06.400921 706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:07.808333 705 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 700
W20250810 20:28:06.401021 707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:08.233011 700 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.833s user 0.647s sys 1.185s
W20250810 20:28:08.234100 700 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.834s user 0.648s sys 1.186s
W20250810 20:28:08.236092 709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:08.239342 708 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1832 milliseconds
I20250810 20:28:08.239377 700 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:08.240773 700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:08.243455 700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:08.244904 700 hybrid_clock.cc:648] HybridClock initialized: now 1754857688244891 us; error 102 us; skew 500 ppm
I20250810 20:28:08.245787 700 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:08.252291 700 webserver.cc:489] Webserver started at http://127.30.243.254:44749/ using document root <none> and password file <none>
I20250810 20:28:08.253299 700 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:08.253551 700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:08.254030 700 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:08.258694 700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "54f242f8b6a24281889fc036f9a9cfc9"
format_stamp: "Formatted at 2025-08-10 20:28:08 on dist-test-slave-7swh"
I20250810 20:28:08.260036 700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "54f242f8b6a24281889fc036f9a9cfc9"
format_stamp: "Formatted at 2025-08-10 20:28:08 on dist-test-slave-7swh"
I20250810 20:28:08.267851 700 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20250810 20:28:08.273790 716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:08.274907 700 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250810 20:28:08.275303 700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "54f242f8b6a24281889fc036f9a9cfc9"
format_stamp: "Formatted at 2025-08-10 20:28:08 on dist-test-slave-7swh"
I20250810 20:28:08.275655 700 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:08.330538 700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:08.332130 700 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:08.332592 700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:08.408243 700 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:41319
I20250810 20:28:08.408432 767 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:41319 every 8 connection(s)
I20250810 20:28:08.411136 700 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 20:28:08.415197 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 700
I20250810 20:28:08.415700 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 20:28:08.417165 768 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:08.443499 768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9: Bootstrap starting.
I20250810 20:28:08.449453 768 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:08.451242 768 log.cc:826] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:08.455965 768 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9: No bootstrap required, opened a new log
I20250810 20:28:08.474568 768 raft_consensus.cc:357] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f242f8b6a24281889fc036f9a9cfc9" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 41319 } }
I20250810 20:28:08.475307 768 raft_consensus.cc:383] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:08.475518 768 raft_consensus.cc:738] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 54f242f8b6a24281889fc036f9a9cfc9, State: Initialized, Role: FOLLOWER
I20250810 20:28:08.476227 768 consensus_queue.cc:260] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f242f8b6a24281889fc036f9a9cfc9" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 41319 } }
I20250810 20:28:08.476742 768 raft_consensus.cc:397] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:28:08.476995 768 raft_consensus.cc:491] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:28:08.477289 768 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:08.482059 768 raft_consensus.cc:513] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f242f8b6a24281889fc036f9a9cfc9" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 41319 } }
I20250810 20:28:08.482852 768 leader_election.cc:304] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 54f242f8b6a24281889fc036f9a9cfc9; no voters:
I20250810 20:28:08.484596 768 leader_election.cc:290] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:28:08.485445 773 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:28:08.487818 773 raft_consensus.cc:695] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [term 1 LEADER]: Becoming Leader. State: Replica: 54f242f8b6a24281889fc036f9a9cfc9, State: Running, Role: LEADER
I20250810 20:28:08.488571 773 consensus_queue.cc:237] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f242f8b6a24281889fc036f9a9cfc9" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 41319 } }
I20250810 20:28:08.489043 768 sys_catalog.cc:564] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:28:08.499051 775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 54f242f8b6a24281889fc036f9a9cfc9. Latest consensus state: current_term: 1 leader_uuid: "54f242f8b6a24281889fc036f9a9cfc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f242f8b6a24281889fc036f9a9cfc9" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 41319 } } }
I20250810 20:28:08.499285 774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "54f242f8b6a24281889fc036f9a9cfc9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "54f242f8b6a24281889fc036f9a9cfc9" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 41319 } } }
I20250810 20:28:08.500113 775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [sys.catalog]: This master's current role is: LEADER
I20250810 20:28:08.500090 774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9 [sys.catalog]: This master's current role is: LEADER
I20250810 20:28:08.507239 781 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:28:08.521169 781 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:28:08.539203 781 catalog_manager.cc:1349] Generated new cluster ID: 7e1902ba907d40e8b271c36489f49bd6
I20250810 20:28:08.539546 781 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:28:08.565939 781 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 20:28:08.568094 781 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:28:08.584456 781 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 54f242f8b6a24281889fc036f9a9cfc9: Generated new TSK 0
I20250810 20:28:08.585755 781 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 20:28:08.597069 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:0
--local_ip_for_outbound_sockets=127.30.243.193
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:41319
--builtin_ntp_servers=127.30.243.212:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250810 20:28:08.928577 792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:08.929160 792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:08.929668 792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:08.963992 792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:08.964900 792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:28:09.002506 792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:41319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:09.003968 792 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:09.005589 792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:09.019793 798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:10.421927 797 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 792
W20250810 20:28:10.516325 792 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.495s user 0.510s sys 0.828s
W20250810 20:28:10.518151 792 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.497s user 0.511s sys 0.829s
W20250810 20:28:10.518441 800 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1496 milliseconds
I20250810 20:28:10.519004 792 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250810 20:28:10.519057 801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:09.021270 799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:10.525085 792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:10.527940 792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:10.529484 792 hybrid_clock.cc:648] HybridClock initialized: now 1754857690529447 us; error 90 us; skew 500 ppm
I20250810 20:28:10.530306 792 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:10.537181 792 webserver.cc:489] Webserver started at http://127.30.243.193:36159/ using document root <none> and password file <none>
I20250810 20:28:10.538213 792 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:10.538432 792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:10.538985 792 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:10.543617 792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c902787ced1e42648dcc3cb3819dcf18"
format_stamp: "Formatted at 2025-08-10 20:28:10 on dist-test-slave-7swh"
I20250810 20:28:10.544860 792 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c902787ced1e42648dcc3cb3819dcf18"
format_stamp: "Formatted at 2025-08-10 20:28:10 on dist-test-slave-7swh"
I20250810 20:28:10.560613 792 fs_manager.cc:696] Time spent creating directory manager: real 0.015s user 0.009s sys 0.000s
I20250810 20:28:10.567258 809 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:10.568614 792 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250810 20:28:10.569000 792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c902787ced1e42648dcc3cb3819dcf18"
format_stamp: "Formatted at 2025-08-10 20:28:10 on dist-test-slave-7swh"
I20250810 20:28:10.569375 792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:10.630100 792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:10.631733 792 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:10.632195 792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:10.634845 792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:28:10.639683 792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:28:10.639921 792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:10.640183 792 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:28:10.640345 792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:10.826500 792 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:44367
I20250810 20:28:10.826611 921 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:44367 every 8 connection(s)
I20250810 20:28:10.830430 792 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 20:28:10.839257 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 792
I20250810 20:28:10.840006 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 20:28:10.849464 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.194:0
--local_ip_for_outbound_sockets=127.30.243.194
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:41319
--builtin_ntp_servers=127.30.243.212:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:28:10.863492 922 heartbeater.cc:344] Connected to a master server at 127.30.243.254:41319
I20250810 20:28:10.864068 922 heartbeater.cc:461] Registering TS with master...
I20250810 20:28:10.865514 922 heartbeater.cc:507] Master 127.30.243.254:41319 requested a full tablet report, sending...
I20250810 20:28:10.869179 733 ts_manager.cc:194] Registered new tserver with Master: c902787ced1e42648dcc3cb3819dcf18 (127.30.243.193:44367)
I20250810 20:28:10.872606 733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:56941
W20250810 20:28:11.182044 926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:11.182575 926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:11.183168 926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:11.216917 926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:11.217854 926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.194
I20250810 20:28:11.254235 926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:41319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.194
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:11.255715 926 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:11.257514 926 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:11.270258 932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:11.877763 922 heartbeater.cc:499] Master 127.30.243.254:41319 was elected leader, sending a full tablet report...
W20250810 20:28:11.271615 933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:11.275710 935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:12.467662 934 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250810 20:28:12.467818 926 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:12.471414 926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:12.474797 926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:12.476313 926 hybrid_clock.cc:648] HybridClock initialized: now 1754857692476266 us; error 70 us; skew 500 ppm
I20250810 20:28:12.477159 926 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:12.484115 926 webserver.cc:489] Webserver started at http://127.30.243.194:44713/ using document root <none> and password file <none>
I20250810 20:28:12.485105 926 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:12.485306 926 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:12.485818 926 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:12.490419 926 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "73b1bf723b5f4be8bd5d1ed676838c5b"
format_stamp: "Formatted at 2025-08-10 20:28:12 on dist-test-slave-7swh"
I20250810 20:28:12.491633 926 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "73b1bf723b5f4be8bd5d1ed676838c5b"
format_stamp: "Formatted at 2025-08-10 20:28:12 on dist-test-slave-7swh"
I20250810 20:28:12.498966 926 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250810 20:28:12.505084 942 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:12.506211 926 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250810 20:28:12.506560 926 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "73b1bf723b5f4be8bd5d1ed676838c5b"
format_stamp: "Formatted at 2025-08-10 20:28:12 on dist-test-slave-7swh"
I20250810 20:28:12.506896 926 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:12.563761 926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:12.565294 926 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:12.565735 926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:12.568265 926 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:28:12.572737 926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:28:12.572968 926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:12.573254 926 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:28:12.573408 926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:12.713337 926 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.194:46735
I20250810 20:28:12.713456 1054 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.194:46735 every 8 connection(s)
I20250810 20:28:12.715972 926 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 20:28:12.718531 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 926
I20250810 20:28:12.719188 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 20:28:12.726155 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.195:0
--local_ip_for_outbound_sockets=127.30.243.195
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:41319
--builtin_ntp_servers=127.30.243.212:46849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:28:12.739557 1055 heartbeater.cc:344] Connected to a master server at 127.30.243.254:41319
I20250810 20:28:12.740406 1055 heartbeater.cc:461] Registering TS with master...
I20250810 20:28:12.742008 1055 heartbeater.cc:507] Master 127.30.243.254:41319 requested a full tablet report, sending...
I20250810 20:28:12.745115 733 ts_manager.cc:194] Registered new tserver with Master: 73b1bf723b5f4be8bd5d1ed676838c5b (127.30.243.194:46735)
I20250810 20:28:12.746431 733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.194:48879
W20250810 20:28:13.048882 1059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:13.049435 1059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:13.049984 1059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:13.083082 1059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:13.084095 1059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.195
I20250810 20:28:13.121451 1059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:46849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:41319
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.195
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:13.122822 1059 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:13.124451 1059 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:13.137259 1065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:13.751073 1055 heartbeater.cc:499] Master 127.30.243.254:41319 was elected leader, sending a full tablet report...
W20250810 20:28:13.137326 1066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:14.410902 1059 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.273s user 0.440s sys 0.825s
W20250810 20:28:14.411586 1059 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.274s user 0.440s sys 0.829s
W20250810 20:28:14.412335 1067 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1275 milliseconds
W20250810 20:28:14.414259 1068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:14.414332 1059 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:14.415899 1059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:14.419512 1059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:14.421053 1059 hybrid_clock.cc:648] HybridClock initialized: now 1754857694420977 us; error 98 us; skew 500 ppm
I20250810 20:28:14.421922 1059 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:14.431761 1059 webserver.cc:489] Webserver started at http://127.30.243.195:44215/ using document root <none> and password file <none>
I20250810 20:28:14.432819 1059 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:14.433079 1059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:14.433568 1059 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:14.438409 1059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ad9b99b31ecf45358f1265441076a133"
format_stamp: "Formatted at 2025-08-10 20:28:14 on dist-test-slave-7swh"
I20250810 20:28:14.439801 1059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ad9b99b31ecf45358f1265441076a133"
format_stamp: "Formatted at 2025-08-10 20:28:14 on dist-test-slave-7swh"
I20250810 20:28:14.448170 1059 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20250810 20:28:14.455695 1075 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:14.457100 1059 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250810 20:28:14.457469 1059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ad9b99b31ecf45358f1265441076a133"
format_stamp: "Formatted at 2025-08-10 20:28:14 on dist-test-slave-7swh"
I20250810 20:28:14.457828 1059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:14.526288 1059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:14.528115 1059 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:14.528637 1059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:14.531366 1059 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:28:14.535902 1059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:28:14.536159 1059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:14.536422 1059 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:28:14.536583 1059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:14.679448 1059 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.195:34597
I20250810 20:28:14.679541 1187 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.195:34597 every 8 connection(s)
I20250810 20:28:14.682247 1059 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 20:28:14.692893 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 1059
I20250810 20:28:14.693424 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 20:28:14.705935 1188 heartbeater.cc:344] Connected to a master server at 127.30.243.254:41319
I20250810 20:28:14.706416 1188 heartbeater.cc:461] Registering TS with master...
I20250810 20:28:14.707492 1188 heartbeater.cc:507] Master 127.30.243.254:41319 requested a full tablet report, sending...
I20250810 20:28:14.709915 733 ts_manager.cc:194] Registered new tserver with Master: ad9b99b31ecf45358f1265441076a133 (127.30.243.195:34597)
I20250810 20:28:14.711802 733 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.195:38759
I20250810 20:28:14.715330 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:28:14.758203 733 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58002:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250810 20:28:14.778208 733 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 20:28:14.844890 857 tablet_service.cc:1468] Processing CreateTablet for tablet 0088787d941949b18e121e2cd92fca78 (DEFAULT_TABLE table=TestTable [id=7abc5533bb02420ba4305e8c73db10c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:28:14.846856 857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0088787d941949b18e121e2cd92fca78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:14.846748 990 tablet_service.cc:1468] Processing CreateTablet for tablet 0088787d941949b18e121e2cd92fca78 (DEFAULT_TABLE table=TestTable [id=7abc5533bb02420ba4305e8c73db10c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:28:14.848944 990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0088787d941949b18e121e2cd92fca78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:14.850430 1123 tablet_service.cc:1468] Processing CreateTablet for tablet 0088787d941949b18e121e2cd92fca78 (DEFAULT_TABLE table=TestTable [id=7abc5533bb02420ba4305e8c73db10c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:28:14.852485 1123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0088787d941949b18e121e2cd92fca78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:14.874145 1207 tablet_bootstrap.cc:492] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18: Bootstrap starting.
I20250810 20:28:14.878556 1208 tablet_bootstrap.cc:492] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b: Bootstrap starting.
I20250810 20:28:14.882120 1207 tablet_bootstrap.cc:654] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:14.883479 1209 tablet_bootstrap.cc:492] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133: Bootstrap starting.
I20250810 20:28:14.886791 1208 tablet_bootstrap.cc:654] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:14.887368 1207 log.cc:826] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:14.889580 1208 log.cc:826] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:14.890997 1209 tablet_bootstrap.cc:654] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:14.893036 1209 log.cc:826] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:14.894407 1207 tablet_bootstrap.cc:492] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18: No bootstrap required, opened a new log
I20250810 20:28:14.895740 1207 ts_tablet_manager.cc:1397] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.004s
I20250810 20:28:14.903146 1209 tablet_bootstrap.cc:492] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133: No bootstrap required, opened a new log
I20250810 20:28:14.903267 1208 tablet_bootstrap.cc:492] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b: No bootstrap required, opened a new log
I20250810 20:28:14.903764 1208 ts_tablet_manager.cc:1397] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b: Time spent bootstrapping tablet: real 0.026s user 0.018s sys 0.004s
I20250810 20:28:14.903764 1209 ts_tablet_manager.cc:1397] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.014s
I20250810 20:28:14.917412 1207 raft_consensus.cc:357] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:14.918160 1207 raft_consensus.cc:383] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:14.918452 1207 raft_consensus.cc:738] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902787ced1e42648dcc3cb3819dcf18, State: Initialized, Role: FOLLOWER
I20250810 20:28:14.919250 1207 consensus_queue.cc:260] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:14.927390 1207 ts_tablet_manager.cc:1428] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18: Time spent starting tablet: real 0.031s user 0.022s sys 0.009s
I20250810 20:28:14.931793 1209 raft_consensus.cc:357] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:14.931799 1208 raft_consensus.cc:357] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:14.932744 1208 raft_consensus.cc:383] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:14.932875 1209 raft_consensus.cc:383] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:14.933115 1208 raft_consensus.cc:738] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b1bf723b5f4be8bd5d1ed676838c5b, State: Initialized, Role: FOLLOWER
I20250810 20:28:14.933218 1209 raft_consensus.cc:738] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9b99b31ecf45358f1265441076a133, State: Initialized, Role: FOLLOWER
I20250810 20:28:14.934108 1209 consensus_queue.cc:260] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:14.934162 1208 consensus_queue.cc:260] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:14.937654 1188 heartbeater.cc:499] Master 127.30.243.254:41319 was elected leader, sending a full tablet report...
I20250810 20:28:14.939778 1209 ts_tablet_manager.cc:1428] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133: Time spent starting tablet: real 0.036s user 0.030s sys 0.005s
I20250810 20:28:14.942294 1208 ts_tablet_manager.cc:1428] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b: Time spent starting tablet: real 0.038s user 0.032s sys 0.004s
W20250810 20:28:14.973290 1056 tablet.cc:2378] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 20:28:15.099056 923 tablet.cc:2378] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 20:28:15.187330 1189 tablet.cc:2378] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:28:15.221275 1215 raft_consensus.cc:491] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:28:15.221776 1215 raft_consensus.cc:513] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.224346 1215 leader_election.cc:290] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902787ced1e42648dcc3cb3819dcf18 (127.30.243.193:44367), ad9b99b31ecf45358f1265441076a133 (127.30.243.195:34597)
I20250810 20:28:15.237421 877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0088787d941949b18e121e2cd92fca78" candidate_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902787ced1e42648dcc3cb3819dcf18" is_pre_election: true
I20250810 20:28:15.238299 877 raft_consensus.cc:2466] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73b1bf723b5f4be8bd5d1ed676838c5b in term 0.
I20250810 20:28:15.238716 1143 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0088787d941949b18e121e2cd92fca78" candidate_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9b99b31ecf45358f1265441076a133" is_pre_election: true
I20250810 20:28:15.239503 1143 raft_consensus.cc:2466] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73b1bf723b5f4be8bd5d1ed676838c5b in term 0.
I20250810 20:28:15.239552 946 leader_election.cc:304] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b1bf723b5f4be8bd5d1ed676838c5b, c902787ced1e42648dcc3cb3819dcf18; no voters:
I20250810 20:28:15.240608 1215 raft_consensus.cc:2802] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 20:28:15.241016 1215 raft_consensus.cc:491] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 20:28:15.241290 1215 raft_consensus.cc:3058] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:15.246150 1215 raft_consensus.cc:513] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.247650 1215 leader_election.cc:290] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [CANDIDATE]: Term 1 election: Requested vote from peers c902787ced1e42648dcc3cb3819dcf18 (127.30.243.193:44367), ad9b99b31ecf45358f1265441076a133 (127.30.243.195:34597)
I20250810 20:28:15.249585 1143 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0088787d941949b18e121e2cd92fca78" candidate_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9b99b31ecf45358f1265441076a133"
I20250810 20:28:15.249585 877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0088787d941949b18e121e2cd92fca78" candidate_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902787ced1e42648dcc3cb3819dcf18"
I20250810 20:28:15.250170 877 raft_consensus.cc:3058] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:15.250173 1143 raft_consensus.cc:3058] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:15.257735 877 raft_consensus.cc:2466] T 0088787d941949b18e121e2cd92fca78 P c902787ced1e42648dcc3cb3819dcf18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73b1bf723b5f4be8bd5d1ed676838c5b in term 1.
I20250810 20:28:15.257728 1143 raft_consensus.cc:2466] T 0088787d941949b18e121e2cd92fca78 P ad9b99b31ecf45358f1265441076a133 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73b1bf723b5f4be8bd5d1ed676838c5b in term 1.
I20250810 20:28:15.259173 946 leader_election.cc:304] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b1bf723b5f4be8bd5d1ed676838c5b, ad9b99b31ecf45358f1265441076a133; no voters:
I20250810 20:28:15.259855 1215 raft_consensus.cc:2802] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:28:15.261518 1215 raft_consensus.cc:695] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 1 LEADER]: Becoming Leader. State: Replica: 73b1bf723b5f4be8bd5d1ed676838c5b, State: Running, Role: LEADER
I20250810 20:28:15.262321 1215 consensus_queue.cc:237] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.274169 733 catalog_manager.cc:5582] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 73b1bf723b5f4be8bd5d1ed676838c5b (127.30.243.194). New cstate: current_term: 1 leader_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } health_report { overall_health: UNKNOWN } } }
I20250810 20:28:15.296722 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:28:15.300273 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c902787ced1e42648dcc3cb3819dcf18 to finish bootstrapping
I20250810 20:28:15.313503 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 73b1bf723b5f4be8bd5d1ed676838c5b to finish bootstrapping
I20250810 20:28:15.325639 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ad9b99b31ecf45358f1265441076a133 to finish bootstrapping
I20250810 20:28:15.341609 733 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58002:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250810 20:28:15.343436 733 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 20:28:15.361501 857 tablet_service.cc:1468] Processing CreateTablet for tablet d093f48824934128878f1b284334e6d5 (DEFAULT_TABLE table=TestAnotherTable [id=450f41cb377443d1b9dcf13b2819b71e]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250810 20:28:15.362056 990 tablet_service.cc:1468] Processing CreateTablet for tablet d093f48824934128878f1b284334e6d5 (DEFAULT_TABLE table=TestAnotherTable [id=450f41cb377443d1b9dcf13b2819b71e]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250810 20:28:15.362689 857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d093f48824934128878f1b284334e6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:15.363170 990 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d093f48824934128878f1b284334e6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:15.363301 1123 tablet_service.cc:1468] Processing CreateTablet for tablet d093f48824934128878f1b284334e6d5 (DEFAULT_TABLE table=TestAnotherTable [id=450f41cb377443d1b9dcf13b2819b71e]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250810 20:28:15.364414 1123 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d093f48824934128878f1b284334e6d5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:15.375751 1207 tablet_bootstrap.cc:492] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18: Bootstrap starting.
I20250810 20:28:15.378528 1208 tablet_bootstrap.cc:492] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b: Bootstrap starting.
I20250810 20:28:15.380399 1209 tablet_bootstrap.cc:492] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133: Bootstrap starting.
I20250810 20:28:15.383333 1207 tablet_bootstrap.cc:654] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:15.385192 1208 tablet_bootstrap.cc:654] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:15.387105 1209 tablet_bootstrap.cc:654] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:15.392032 1207 tablet_bootstrap.cc:492] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18: No bootstrap required, opened a new log
I20250810 20:28:15.392143 1208 tablet_bootstrap.cc:492] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b: No bootstrap required, opened a new log
I20250810 20:28:15.392506 1208 ts_tablet_manager.cc:1397] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20250810 20:28:15.392719 1207 ts_tablet_manager.cc:1397] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.008s
I20250810 20:28:15.393664 1209 tablet_bootstrap.cc:492] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133: No bootstrap required, opened a new log
I20250810 20:28:15.394152 1209 ts_tablet_manager.cc:1397] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.001s
I20250810 20:28:15.395058 1208 raft_consensus.cc:357] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.395742 1208 raft_consensus.cc:383] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:15.396040 1208 raft_consensus.cc:738] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73b1bf723b5f4be8bd5d1ed676838c5b, State: Initialized, Role: FOLLOWER
I20250810 20:28:15.395709 1207 raft_consensus.cc:357] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.396404 1207 raft_consensus.cc:383] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:15.396696 1207 raft_consensus.cc:738] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c902787ced1e42648dcc3cb3819dcf18, State: Initialized, Role: FOLLOWER
I20250810 20:28:15.396760 1208 consensus_queue.cc:260] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.396817 1209 raft_consensus.cc:357] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.397516 1209 raft_consensus.cc:383] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:15.397804 1209 raft_consensus.cc:738] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad9b99b31ecf45358f1265441076a133, State: Initialized, Role: FOLLOWER
I20250810 20:28:15.397360 1207 consensus_queue.cc:260] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.398490 1209 consensus_queue.cc:260] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.400192 1208 ts_tablet_manager.cc:1428] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20250810 20:28:15.402621 1207 ts_tablet_manager.cc:1428] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18: Time spent starting tablet: real 0.009s user 0.007s sys 0.000s
I20250810 20:28:15.403824 1209 ts_tablet_manager.cc:1428] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20250810 20:28:15.646438 1215 raft_consensus.cc:491] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:28:15.646914 1215 raft_consensus.cc:513] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.648365 1215 leader_election.cc:290] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c902787ced1e42648dcc3cb3819dcf18 (127.30.243.193:44367), ad9b99b31ecf45358f1265441076a133 (127.30.243.195:34597)
I20250810 20:28:15.649395 877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d093f48824934128878f1b284334e6d5" candidate_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902787ced1e42648dcc3cb3819dcf18" is_pre_election: true
I20250810 20:28:15.649561 1143 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d093f48824934128878f1b284334e6d5" candidate_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9b99b31ecf45358f1265441076a133" is_pre_election: true
I20250810 20:28:15.649891 877 raft_consensus.cc:2466] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73b1bf723b5f4be8bd5d1ed676838c5b in term 0.
I20250810 20:28:15.650141 1143 raft_consensus.cc:2466] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73b1bf723b5f4be8bd5d1ed676838c5b in term 0.
I20250810 20:28:15.650815 946 leader_election.cc:304] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b1bf723b5f4be8bd5d1ed676838c5b, c902787ced1e42648dcc3cb3819dcf18; no voters:
I20250810 20:28:15.651497 1215 raft_consensus.cc:2802] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 20:28:15.651746 1215 raft_consensus.cc:491] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 20:28:15.652011 1215 raft_consensus.cc:3058] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:15.656147 1215 raft_consensus.cc:513] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.658206 1215 leader_election.cc:290] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [CANDIDATE]: Term 1 election: Requested vote from peers c902787ced1e42648dcc3cb3819dcf18 (127.30.243.193:44367), ad9b99b31ecf45358f1265441076a133 (127.30.243.195:34597)
I20250810 20:28:15.658829 877 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d093f48824934128878f1b284334e6d5" candidate_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c902787ced1e42648dcc3cb3819dcf18"
I20250810 20:28:15.659204 1143 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d093f48824934128878f1b284334e6d5" candidate_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad9b99b31ecf45358f1265441076a133"
I20250810 20:28:15.659386 877 raft_consensus.cc:3058] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:15.659693 1143 raft_consensus.cc:3058] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:15.663583 877 raft_consensus.cc:2466] T d093f48824934128878f1b284334e6d5 P c902787ced1e42648dcc3cb3819dcf18 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73b1bf723b5f4be8bd5d1ed676838c5b in term 1.
I20250810 20:28:15.663831 1143 raft_consensus.cc:2466] T d093f48824934128878f1b284334e6d5 P ad9b99b31ecf45358f1265441076a133 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73b1bf723b5f4be8bd5d1ed676838c5b in term 1.
I20250810 20:28:15.664557 946 leader_election.cc:304] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 73b1bf723b5f4be8bd5d1ed676838c5b, c902787ced1e42648dcc3cb3819dcf18; no voters:
I20250810 20:28:15.665221 1215 raft_consensus.cc:2802] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:28:15.665657 1215 raft_consensus.cc:695] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [term 1 LEADER]: Becoming Leader. State: Replica: 73b1bf723b5f4be8bd5d1ed676838c5b, State: Running, Role: LEADER
I20250810 20:28:15.666404 1215 consensus_queue.cc:237] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } }
I20250810 20:28:15.672948 733 catalog_manager.cc:5582] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b reported cstate change: term changed from 0 to 1, leader changed from <none> to 73b1bf723b5f4be8bd5d1ed676838c5b (127.30.243.194). New cstate: current_term: 1 leader_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73b1bf723b5f4be8bd5d1ed676838c5b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46735 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 } health_report { overall_health: UNKNOWN } } }
I20250810 20:28:15.760226 1215 consensus_queue.cc:1035] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 20:28:15.788677 1215 consensus_queue.cc:1035] T 0088787d941949b18e121e2cd92fca78 P 73b1bf723b5f4be8bd5d1ed676838c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250810 20:28:16.030802 1225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:16.031497 1225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:16.065991 1225 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250810 20:28:16.122205 1218 consensus_queue.cc:1035] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad9b99b31ecf45358f1265441076a133" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 34597 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250810 20:28:16.130704 1218 consensus_queue.cc:1035] T d093f48824934128878f1b284334e6d5 P 73b1bf723b5f4be8bd5d1ed676838c5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c902787ced1e42648dcc3cb3819dcf18" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250810 20:28:17.459405 1225 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.336s user 0.497s sys 0.792s
W20250810 20:28:17.459802 1225 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.336s user 0.497s sys 0.792s
W20250810 20:28:18.874248 1248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:18.874893 1248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:18.910144 1248 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250810 20:28:20.235296 1248 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.283s user 0.485s sys 0.797s
W20250810 20:28:20.235747 1248 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.284s user 0.485s sys 0.797s
W20250810 20:28:21.639588 1265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:21.640166 1265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:21.672112 1265 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250810 20:28:22.858224 764 debug-util.cc:398] Leaking SignalData structure 0x7b08000a97e0 after lost signal to thread 701
W20250810 20:28:22.859002 764 debug-util.cc:398] Leaking SignalData structure 0x7b08000ab3a0 after lost signal to thread 767
W20250810 20:28:23.122202 1270 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 1265
W20250810 20:28:23.122972 1270 kernel_stack_watchdog.cc:198] Thread 1265 stuck at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:642 for 404ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 20:28:23.145434 1265 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.429s user 0.541s sys 0.881s
W20250810 20:28:23.270998 1265 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.555s user 0.550s sys 0.889s
W20250810 20:28:24.705173 1279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:24.705796 1279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:24.739858 1279 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250810 20:28:26.061209 1279 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.279s user 0.502s sys 0.775s
W20250810 20:28:26.061625 1279 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.280s user 0.502s sys 0.775s
I20250810 20:28:27.144651 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 792
I20250810 20:28:27.172149 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 926
I20250810 20:28:27.202088 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 1059
I20250810 20:28:27.230152 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 700
2025-08-10T20:28:27Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (21369 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250810 20:28:27.296670 31695 test_util.cc:276] Using random seed: -1505339519
I20250810 20:28:27.300902 31695 ts_itest-base.cc:115] Starting cluster with:
I20250810 20:28:27.301090 31695 ts_itest-base.cc:116] --------------
I20250810 20:28:27.301252 31695 ts_itest-base.cc:117] 3 tablet servers
I20250810 20:28:27.301394 31695 ts_itest-base.cc:118] 3 replicas per TS
I20250810 20:28:27.301542 31695 ts_itest-base.cc:119] --------------
2025-08-10T20:28:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T20:28:27Z Disabled control of system clock
I20250810 20:28:27.341243 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:43185
--webserver_interface=127.30.243.254
--webserver_port=0
--builtin_ntp_servers=127.30.243.212:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:43185
--superuser_acl=no-such-user with env {}
W20250810 20:28:27.657306 1300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:27.657907 1300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:27.658412 1300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:27.690711 1300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:28:27.691103 1300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:27.691370 1300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:28:27.691617 1300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:28:27.728139 1300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:43185
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:43185
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:27.729525 1300 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:27.731483 1300 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:27.743988 1306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:27.744549 1307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:27.748274 1309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:28.966697 1308 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1220 milliseconds
I20250810 20:28:28.967101 1300 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:28.968376 1300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:28.971752 1300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:28.973310 1300 hybrid_clock.cc:648] HybridClock initialized: now 1754857708973227 us; error 93 us; skew 500 ppm
I20250810 20:28:28.974216 1300 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:28.981465 1300 webserver.cc:489] Webserver started at http://127.30.243.254:39145/ using document root <none> and password file <none>
I20250810 20:28:28.982493 1300 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:28.982749 1300 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:28.983278 1300 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:28.989784 1300 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0fa69eb90b374906a58a83518da74efa"
format_stamp: "Formatted at 2025-08-10 20:28:28 on dist-test-slave-7swh"
I20250810 20:28:28.991063 1300 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0fa69eb90b374906a58a83518da74efa"
format_stamp: "Formatted at 2025-08-10 20:28:28 on dist-test-slave-7swh"
I20250810 20:28:28.999033 1300 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250810 20:28:29.005374 1316 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:29.006533 1300 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250810 20:28:29.006904 1300 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0fa69eb90b374906a58a83518da74efa"
format_stamp: "Formatted at 2025-08-10 20:28:28 on dist-test-slave-7swh"
I20250810 20:28:29.007305 1300 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:29.067440 1300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:29.068989 1300 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:29.069430 1300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:29.144793 1300 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:43185
I20250810 20:28:29.144865 1367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:43185 every 8 connection(s)
I20250810 20:28:29.147676 1300 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 20:28:29.152379 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 1300
I20250810 20:28:29.152814 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 20:28:29.154076 1368 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:29.180131 1368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa: Bootstrap starting.
I20250810 20:28:29.186967 1368 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:29.189128 1368 log.cc:826] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:29.193617 1368 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa: No bootstrap required, opened a new log
I20250810 20:28:29.211858 1368 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa69eb90b374906a58a83518da74efa" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 43185 } }
I20250810 20:28:29.212546 1368 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:29.212755 1368 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fa69eb90b374906a58a83518da74efa, State: Initialized, Role: FOLLOWER
I20250810 20:28:29.213389 1368 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa69eb90b374906a58a83518da74efa" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 43185 } }
I20250810 20:28:29.213892 1368 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:28:29.214128 1368 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:28:29.214404 1368 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:29.218812 1368 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa69eb90b374906a58a83518da74efa" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 43185 } }
I20250810 20:28:29.219568 1368 leader_election.cc:304] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fa69eb90b374906a58a83518da74efa; no voters:
I20250810 20:28:29.221194 1368 leader_election.cc:290] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:28:29.222061 1373 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:28:29.224296 1373 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [term 1 LEADER]: Becoming Leader. State: Replica: 0fa69eb90b374906a58a83518da74efa, State: Running, Role: LEADER
I20250810 20:28:29.225333 1368 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:28:29.225241 1373 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa69eb90b374906a58a83518da74efa" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 43185 } }
I20250810 20:28:29.237733 1375 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fa69eb90b374906a58a83518da74efa. Latest consensus state: current_term: 1 leader_uuid: "0fa69eb90b374906a58a83518da74efa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa69eb90b374906a58a83518da74efa" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 43185 } } }
I20250810 20:28:29.237963 1374 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fa69eb90b374906a58a83518da74efa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fa69eb90b374906a58a83518da74efa" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 43185 } } }
I20250810 20:28:29.238677 1374 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [sys.catalog]: This master's current role is: LEADER
I20250810 20:28:29.238683 1375 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa [sys.catalog]: This master's current role is: LEADER
I20250810 20:28:29.246109 1383 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:28:29.258268 1383 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:28:29.275046 1383 catalog_manager.cc:1349] Generated new cluster ID: f7d711cb1d8f442796ca8f5f7ab95ef3
I20250810 20:28:29.275377 1383 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:28:29.314078 1383 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 20:28:29.316172 1383 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:28:29.332149 1383 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0fa69eb90b374906a58a83518da74efa: Generated new TSK 0
I20250810 20:28:29.333278 1383 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 20:28:29.352806 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:0
--local_ip_for_outbound_sockets=127.30.243.193
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:43185
--builtin_ntp_servers=127.30.243.212:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250810 20:28:29.681700 1393 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:29.682217 1393 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:29.682724 1393 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:29.715090 1393 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:29.716133 1393 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:28:29.752089 1393 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:43185
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:29.753669 1393 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:29.755643 1393 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:29.768939 1399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:31.171995 1398 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 1393
W20250810 20:28:31.416617 1393 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.647s user 0.558s sys 1.073s
W20250810 20:28:31.417137 1393 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.647s user 0.558s sys 1.073s
W20250810 20:28:29.770661 1400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:31.418390 1401 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1648 milliseconds
W20250810 20:28:31.419852 1402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:31.419799 1393 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:31.422878 1393 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:31.425802 1393 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:31.427352 1393 hybrid_clock.cc:648] HybridClock initialized: now 1754857711427274 us; error 101 us; skew 500 ppm
I20250810 20:28:31.428503 1393 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:31.436571 1393 webserver.cc:489] Webserver started at http://127.30.243.193:33509/ using document root <none> and password file <none>
I20250810 20:28:31.437943 1393 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:31.438249 1393 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:31.438838 1393 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:31.445578 1393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "af604afd9b904de3937486c29e5e62a5"
format_stamp: "Formatted at 2025-08-10 20:28:31 on dist-test-slave-7swh"
I20250810 20:28:31.447256 1393 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "af604afd9b904de3937486c29e5e62a5"
format_stamp: "Formatted at 2025-08-10 20:28:31 on dist-test-slave-7swh"
I20250810 20:28:31.457207 1393 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.008s sys 0.000s
I20250810 20:28:31.465204 1409 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:31.466516 1393 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20250810 20:28:31.467037 1393 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "af604afd9b904de3937486c29e5e62a5"
format_stamp: "Formatted at 2025-08-10 20:28:31 on dist-test-slave-7swh"
I20250810 20:28:31.467417 1393 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:31.529155 1393 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:31.530659 1393 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:31.531183 1393 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:31.534343 1393 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:28:31.538913 1393 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:28:31.539227 1393 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250810 20:28:31.539494 1393 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:28:31.539654 1393 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:31.718747 1393 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:33913
I20250810 20:28:31.718847 1521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:33913 every 8 connection(s)
I20250810 20:28:31.721544 1393 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 20:28:31.726570 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 1393
I20250810 20:28:31.727147 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 20:28:31.735042 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.194:0
--local_ip_for_outbound_sockets=127.30.243.194
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:43185
--builtin_ntp_servers=127.30.243.212:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:28:31.752051 1522 heartbeater.cc:344] Connected to a master server at 127.30.243.254:43185
I20250810 20:28:31.752538 1522 heartbeater.cc:461] Registering TS with master...
I20250810 20:28:31.753607 1522 heartbeater.cc:507] Master 127.30.243.254:43185 requested a full tablet report, sending...
I20250810 20:28:31.756601 1333 ts_manager.cc:194] Registered new tserver with Master: af604afd9b904de3937486c29e5e62a5 (127.30.243.193:33913)
I20250810 20:28:31.758620 1333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:54851
W20250810 20:28:32.068764 1526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:32.069307 1526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:32.069824 1526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:32.102674 1526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:32.103597 1526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.194
I20250810 20:28:32.139591 1526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:43185
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.194
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:32.140982 1526 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:32.142609 1526 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:32.156026 1532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:32.222496 1364 debug-util.cc:398] Leaking SignalData structure 0x7b08000932c0 after lost signal to thread 1301
I20250810 20:28:32.762488 1522 heartbeater.cc:499] Master 127.30.243.254:43185 was elected leader, sending a full tablet report...
W20250810 20:28:32.156654 1533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:33.558043 1531 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 1526
W20250810 20:28:33.776083 1531 kernel_stack_watchdog.cc:198] Thread 1526 stuck at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 20:28:33.778196 1535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:33.779234 1526 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.624s user 0.467s sys 1.112s
W20250810 20:28:33.779522 1526 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.624s user 0.467s sys 1.112s
W20250810 20:28:33.780380 1534 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1623 milliseconds
I20250810 20:28:33.780411 1526 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:33.781703 1526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:33.783928 1526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:33.785302 1526 hybrid_clock.cc:648] HybridClock initialized: now 1754857713785264 us; error 50 us; skew 500 ppm
I20250810 20:28:33.786106 1526 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:33.793195 1526 webserver.cc:489] Webserver started at http://127.30.243.194:40205/ using document root <none> and password file <none>
I20250810 20:28:33.794276 1526 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:33.794516 1526 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:33.795048 1526 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:33.799952 1526 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "24da155f558a47c48d1a49ae93bb8028"
format_stamp: "Formatted at 2025-08-10 20:28:33 on dist-test-slave-7swh"
I20250810 20:28:33.801136 1526 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "24da155f558a47c48d1a49ae93bb8028"
format_stamp: "Formatted at 2025-08-10 20:28:33 on dist-test-slave-7swh"
I20250810 20:28:33.809651 1526 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.001s sys 0.009s
I20250810 20:28:33.816112 1542 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:33.817289 1526 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250810 20:28:33.817629 1526 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "24da155f558a47c48d1a49ae93bb8028"
format_stamp: "Formatted at 2025-08-10 20:28:33 on dist-test-slave-7swh"
I20250810 20:28:33.817962 1526 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:33.873270 1526 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:33.874974 1526 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:33.875460 1526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:33.878608 1526 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:28:33.883217 1526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:28:33.883435 1526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:33.883745 1526 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:28:33.883936 1526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:34.065416 1526 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.194:43953
I20250810 20:28:34.065625 1654 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.194:43953 every 8 connection(s)
I20250810 20:28:34.068046 1526 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 20:28:34.075809 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 1526
I20250810 20:28:34.076331 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 20:28:34.084215 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.195:0
--local_ip_for_outbound_sockets=127.30.243.195
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:43185
--builtin_ntp_servers=127.30.243.212:35119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:28:34.095018 1655 heartbeater.cc:344] Connected to a master server at 127.30.243.254:43185
I20250810 20:28:34.095525 1655 heartbeater.cc:461] Registering TS with master...
I20250810 20:28:34.096629 1655 heartbeater.cc:507] Master 127.30.243.254:43185 requested a full tablet report, sending...
I20250810 20:28:34.099305 1333 ts_manager.cc:194] Registered new tserver with Master: 24da155f558a47c48d1a49ae93bb8028 (127.30.243.194:43953)
I20250810 20:28:34.100616 1333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.194:43275
W20250810 20:28:34.430140 1659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:34.430658 1659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:34.431232 1659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:34.465416 1659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:34.466333 1659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.195
I20250810 20:28:34.503655 1659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:35119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:43185
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.195
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:34.505168 1659 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:34.507783 1659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:34.522094 1665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:35.104118 1655 heartbeater.cc:499] Master 127.30.243.254:43185 was elected leader, sending a full tablet report...
W20250810 20:28:34.524199 1666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:34.528918 1668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:35.924654 1667 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1397 milliseconds
I20250810 20:28:35.924809 1659 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:35.926015 1659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:35.928821 1659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:35.930287 1659 hybrid_clock.cc:648] HybridClock initialized: now 1754857715930240 us; error 59 us; skew 500 ppm
I20250810 20:28:35.931180 1659 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:35.937673 1659 webserver.cc:489] Webserver started at http://127.30.243.195:37521/ using document root <none> and password file <none>
I20250810 20:28:35.938627 1659 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:35.938840 1659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:35.939373 1659 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:35.943949 1659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "eb40012b347c464cb29a529d11ed207f"
format_stamp: "Formatted at 2025-08-10 20:28:35 on dist-test-slave-7swh"
I20250810 20:28:35.945139 1659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "eb40012b347c464cb29a529d11ed207f"
format_stamp: "Formatted at 2025-08-10 20:28:35 on dist-test-slave-7swh"
I20250810 20:28:35.952651 1659 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20250810 20:28:35.958634 1675 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:35.959718 1659 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250810 20:28:35.960086 1659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "eb40012b347c464cb29a529d11ed207f"
format_stamp: "Formatted at 2025-08-10 20:28:35 on dist-test-slave-7swh"
I20250810 20:28:35.960436 1659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:36.017114 1659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:36.018666 1659 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:36.019181 1659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:36.021713 1659 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:28:36.025939 1659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:28:36.026189 1659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:36.026436 1659 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:28:36.026595 1659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:36.178463 1659 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.195:37917
I20250810 20:28:36.178617 1787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.195:37917 every 8 connection(s)
I20250810 20:28:36.182005 1659 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 20:28:36.183729 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 1659
I20250810 20:28:36.184331 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 20:28:36.214434 1788 heartbeater.cc:344] Connected to a master server at 127.30.243.254:43185
I20250810 20:28:36.215140 1788 heartbeater.cc:461] Registering TS with master...
I20250810 20:28:36.217114 1788 heartbeater.cc:507] Master 127.30.243.254:43185 requested a full tablet report, sending...
I20250810 20:28:36.221011 1333 ts_manager.cc:194] Registered new tserver with Master: eb40012b347c464cb29a529d11ed207f (127.30.243.195:37917)
I20250810 20:28:36.222452 1333 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.195:37443
I20250810 20:28:36.228751 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:28:36.268999 1332 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52062:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250810 20:28:36.287984 1332 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 20:28:36.342402 1723 tablet_service.cc:1468] Processing CreateTablet for tablet 993398fae2864fefb7f1e13aebbf63a2 (DEFAULT_TABLE table=TestTable [id=caa0d4e0b58c4c6fb68052bce600b58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:28:36.344623 1723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993398fae2864fefb7f1e13aebbf63a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:36.350134 1457 tablet_service.cc:1468] Processing CreateTablet for tablet 993398fae2864fefb7f1e13aebbf63a2 (DEFAULT_TABLE table=TestTable [id=caa0d4e0b58c4c6fb68052bce600b58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:28:36.351682 1457 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993398fae2864fefb7f1e13aebbf63a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:36.352722 1590 tablet_service.cc:1468] Processing CreateTablet for tablet 993398fae2864fefb7f1e13aebbf63a2 (DEFAULT_TABLE table=TestTable [id=caa0d4e0b58c4c6fb68052bce600b58c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:28:36.354758 1590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 993398fae2864fefb7f1e13aebbf63a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:36.373979 1807 tablet_bootstrap.cc:492] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5: Bootstrap starting.
I20250810 20:28:36.376729 1808 tablet_bootstrap.cc:492] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f: Bootstrap starting.
I20250810 20:28:36.382201 1807 tablet_bootstrap.cc:654] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:36.384429 1809 tablet_bootstrap.cc:492] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028: Bootstrap starting.
I20250810 20:28:36.384905 1808 tablet_bootstrap.cc:654] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:36.385063 1807 log.cc:826] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:36.387619 1808 log.cc:826] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:36.391510 1807 tablet_bootstrap.cc:492] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5: No bootstrap required, opened a new log
I20250810 20:28:36.391916 1809 tablet_bootstrap.cc:654] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:36.392061 1807 ts_tablet_manager.cc:1397] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20250810 20:28:36.393386 1808 tablet_bootstrap.cc:492] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f: No bootstrap required, opened a new log
I20250810 20:28:36.393966 1808 ts_tablet_manager.cc:1397] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.017s
I20250810 20:28:36.395901 1809 log.cc:826] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:36.401006 1809 tablet_bootstrap.cc:492] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028: No bootstrap required, opened a new log
I20250810 20:28:36.401484 1809 ts_tablet_manager.cc:1397] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.006s
I20250810 20:28:36.420423 1809 raft_consensus.cc:357] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } }
I20250810 20:28:36.421198 1809 raft_consensus.cc:383] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:36.420816 1807 raft_consensus.cc:357] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } }
I20250810 20:28:36.421482 1809 raft_consensus.cc:738] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24da155f558a47c48d1a49ae93bb8028, State: Initialized, Role: FOLLOWER
I20250810 20:28:36.421747 1807 raft_consensus.cc:383] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:36.422048 1807 raft_consensus.cc:738] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af604afd9b904de3937486c29e5e62a5, State: Initialized, Role: FOLLOWER
I20250810 20:28:36.422322 1809 consensus_queue.cc:260] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } }
I20250810 20:28:36.422672 1808 raft_consensus.cc:357] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } }
I20250810 20:28:36.422979 1807 consensus_queue.cc:260] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } }
I20250810 20:28:36.423614 1808 raft_consensus.cc:383] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:36.423928 1808 raft_consensus.cc:738] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb40012b347c464cb29a529d11ed207f, State: Initialized, Role: FOLLOWER
I20250810 20:28:36.426363 1809 ts_tablet_manager.cc:1428] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028: Time spent starting tablet: real 0.025s user 0.021s sys 0.005s
I20250810 20:28:36.425035 1808 consensus_queue.cc:260] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } }
I20250810 20:28:36.429658 1807 ts_tablet_manager.cc:1428] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5: Time spent starting tablet: real 0.037s user 0.032s sys 0.004s
I20250810 20:28:36.431531 1788 heartbeater.cc:499] Master 127.30.243.254:43185 was elected leader, sending a full tablet report...
I20250810 20:28:36.434048 1808 ts_tablet_manager.cc:1428] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f: Time spent starting tablet: real 0.040s user 0.036s sys 0.001s
W20250810 20:28:36.437563 1789 tablet.cc:2378] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 20:28:36.483577 1523 tablet.cc:2378] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 20:28:36.580420 1656 tablet.cc:2378] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:28:36.688396 1813 raft_consensus.cc:491] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:28:36.688944 1813 raft_consensus.cc:513] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } }
I20250810 20:28:36.691473 1813 leader_election.cc:290] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb40012b347c464cb29a529d11ed207f (127.30.243.195:37917), af604afd9b904de3937486c29e5e62a5 (127.30.243.193:33913)
I20250810 20:28:36.695495 1814 raft_consensus.cc:491] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:28:36.696221 1814 raft_consensus.cc:513] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } }
I20250810 20:28:36.699630 1814 leader_election.cc:290] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb40012b347c464cb29a529d11ed207f (127.30.243.195:37917), 24da155f558a47c48d1a49ae93bb8028 (127.30.243.194:43953)
I20250810 20:28:36.712769 1743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "993398fae2864fefb7f1e13aebbf63a2" candidate_uuid: "24da155f558a47c48d1a49ae93bb8028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb40012b347c464cb29a529d11ed207f" is_pre_election: true
I20250810 20:28:36.713819 1743 raft_consensus.cc:2466] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24da155f558a47c48d1a49ae93bb8028 in term 0.
I20250810 20:28:36.715288 1477 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "993398fae2864fefb7f1e13aebbf63a2" candidate_uuid: "24da155f558a47c48d1a49ae93bb8028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af604afd9b904de3937486c29e5e62a5" is_pre_election: true
I20250810 20:28:36.715505 1546 leader_election.cc:304] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24da155f558a47c48d1a49ae93bb8028, eb40012b347c464cb29a529d11ed207f; no voters:
I20250810 20:28:36.716207 1477 raft_consensus.cc:2466] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24da155f558a47c48d1a49ae93bb8028 in term 0.
I20250810 20:28:36.716487 1813 raft_consensus.cc:2802] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 20:28:36.716849 1813 raft_consensus.cc:491] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 20:28:36.717244 1813 raft_consensus.cc:3058] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:36.724933 1813 raft_consensus.cc:513] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } }
I20250810 20:28:36.727952 1743 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "993398fae2864fefb7f1e13aebbf63a2" candidate_uuid: "24da155f558a47c48d1a49ae93bb8028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb40012b347c464cb29a529d11ed207f"
I20250810 20:28:36.728541 1743 raft_consensus.cc:3058] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:36.736562 1743 raft_consensus.cc:2466] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24da155f558a47c48d1a49ae93bb8028 in term 1.
I20250810 20:28:36.737764 1546 leader_election.cc:304] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24da155f558a47c48d1a49ae93bb8028, eb40012b347c464cb29a529d11ed207f; no voters:
I20250810 20:28:36.741478 1742 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "993398fae2864fefb7f1e13aebbf63a2" candidate_uuid: "af604afd9b904de3937486c29e5e62a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb40012b347c464cb29a529d11ed207f" is_pre_election: true
I20250810 20:28:36.742508 1477 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "993398fae2864fefb7f1e13aebbf63a2" candidate_uuid: "24da155f558a47c48d1a49ae93bb8028" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af604afd9b904de3937486c29e5e62a5"
I20250810 20:28:36.743019 1742 raft_consensus.cc:2391] T 993398fae2864fefb7f1e13aebbf63a2 P eb40012b347c464cb29a529d11ed207f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate af604afd9b904de3937486c29e5e62a5 in current term 1: Already voted for candidate 24da155f558a47c48d1a49ae93bb8028 in this term.
I20250810 20:28:36.743257 1477 raft_consensus.cc:3058] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:36.744534 1813 leader_election.cc:290] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [CANDIDATE]: Term 1 election: Requested vote from peers eb40012b347c464cb29a529d11ed207f (127.30.243.195:37917), af604afd9b904de3937486c29e5e62a5 (127.30.243.193:33913)
I20250810 20:28:36.745126 1813 raft_consensus.cc:2802] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:28:36.745596 1813 raft_consensus.cc:695] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [term 1 LEADER]: Becoming Leader. State: Replica: 24da155f558a47c48d1a49ae93bb8028, State: Running, Role: LEADER
I20250810 20:28:36.746850 1813 consensus_queue.cc:237] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } }
I20250810 20:28:36.751340 1477 raft_consensus.cc:2466] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24da155f558a47c48d1a49ae93bb8028 in term 1.
I20250810 20:28:36.754058 1610 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "993398fae2864fefb7f1e13aebbf63a2" candidate_uuid: "af604afd9b904de3937486c29e5e62a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24da155f558a47c48d1a49ae93bb8028" is_pre_election: true
I20250810 20:28:36.756934 1413 leader_election.cc:304] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: af604afd9b904de3937486c29e5e62a5; no voters: 24da155f558a47c48d1a49ae93bb8028, eb40012b347c464cb29a529d11ed207f
I20250810 20:28:36.757711 1814 raft_consensus.cc:2747] T 993398fae2864fefb7f1e13aebbf63a2 P af604afd9b904de3937486c29e5e62a5 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250810 20:28:36.758069 1331 catalog_manager.cc:5582] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24da155f558a47c48d1a49ae93bb8028 (127.30.243.194). New cstate: current_term: 1 leader_uuid: "24da155f558a47c48d1a49ae93bb8028" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24da155f558a47c48d1a49ae93bb8028" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 43953 } health_report { overall_health: HEALTHY } } }
I20250810 20:28:36.810133 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:28:36.813745 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver af604afd9b904de3937486c29e5e62a5 to finish bootstrapping
I20250810 20:28:36.825894 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 24da155f558a47c48d1a49ae93bb8028 to finish bootstrapping
I20250810 20:28:36.837103 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver eb40012b347c464cb29a529d11ed207f to finish bootstrapping
I20250810 20:28:37.134299 1813 consensus_queue.cc:1035] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af604afd9b904de3937486c29e5e62a5" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 20:28:37.151374 1821 consensus_queue.cc:1035] T 993398fae2864fefb7f1e13aebbf63a2 P 24da155f558a47c48d1a49ae93bb8028 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb40012b347c464cb29a529d11ed207f" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250810 20:28:38.513662 1331 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:52082
I20250810 20:28:39.550251 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 1393
I20250810 20:28:39.576854 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 1526
I20250810 20:28:39.603760 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 1659
I20250810 20:28:39.630064 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 1300
2025-08-10T20:28:39Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (12394 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250810 20:28:39.691638 31695 test_util.cc:276] Using random seed: -1492944554
I20250810 20:28:39.696444 31695 ts_itest-base.cc:115] Starting cluster with:
I20250810 20:28:39.696626 31695 ts_itest-base.cc:116] --------------
I20250810 20:28:39.696771 31695 ts_itest-base.cc:117] 3 tablet servers
I20250810 20:28:39.696913 31695 ts_itest-base.cc:118] 3 replicas per TS
I20250810 20:28:39.697077 31695 ts_itest-base.cc:119] --------------
2025-08-10T20:28:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T20:28:39Z Disabled control of system clock
I20250810 20:28:39.736593 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:34401
--webserver_interface=127.30.243.254
--webserver_port=0
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:34401 with env {}
W20250810 20:28:40.046887 1858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:40.047501 1858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:40.047900 1858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:40.080914 1858 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:28:40.081218 1858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:40.081421 1858 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:28:40.081615 1858 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:28:40.118270 1858 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:34401
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:40.119657 1858 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:40.121570 1858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:40.133147 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:41.543893 1863 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 1858
W20250810 20:28:41.960624 1858 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.827s user 0.694s sys 1.128s
W20250810 20:28:40.134014 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:41.961809 1858 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.829s user 0.695s sys 1.128s
W20250810 20:28:41.962569 1866 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1828 milliseconds
W20250810 20:28:41.963128 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:41.963060 1858 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:41.966572 1858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:41.969312 1858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:41.970724 1858 hybrid_clock.cc:648] HybridClock initialized: now 1754857721970668 us; error 59 us; skew 500 ppm
I20250810 20:28:41.971628 1858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:41.978124 1858 webserver.cc:489] Webserver started at http://127.30.243.254:38047/ using document root <none> and password file <none>
I20250810 20:28:41.979161 1858 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:41.979385 1858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:41.979879 1858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:41.984372 1858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f826241eec6f4799bde3030164f1b95c"
format_stamp: "Formatted at 2025-08-10 20:28:41 on dist-test-slave-7swh"
I20250810 20:28:41.985551 1858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f826241eec6f4799bde3030164f1b95c"
format_stamp: "Formatted at 2025-08-10 20:28:41 on dist-test-slave-7swh"
I20250810 20:28:41.993238 1858 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.005s
I20250810 20:28:41.999195 1874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:42.000324 1858 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250810 20:28:42.000684 1858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f826241eec6f4799bde3030164f1b95c"
format_stamp: "Formatted at 2025-08-10 20:28:41 on dist-test-slave-7swh"
I20250810 20:28:42.001048 1858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:42.062454 1858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:42.064039 1858 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:42.064518 1858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:42.138573 1858 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:34401
I20250810 20:28:42.138643 1925 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:34401 every 8 connection(s)
I20250810 20:28:42.141456 1858 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 20:28:42.142556 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 1858
I20250810 20:28:42.143044 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 20:28:42.148445 1926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:42.173795 1926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Bootstrap starting.
I20250810 20:28:42.179489 1926 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:42.181532 1926 log.cc:826] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:42.186064 1926 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: No bootstrap required, opened a new log
I20250810 20:28:42.204181 1926 raft_consensus.cc:357] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:28:42.204926 1926 raft_consensus.cc:383] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:42.205169 1926 raft_consensus.cc:738] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f826241eec6f4799bde3030164f1b95c, State: Initialized, Role: FOLLOWER
I20250810 20:28:42.205884 1926 consensus_queue.cc:260] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:28:42.206497 1926 raft_consensus.cc:397] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:28:42.206866 1926 raft_consensus.cc:491] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:28:42.207286 1926 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:42.211656 1926 raft_consensus.cc:513] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:28:42.212445 1926 leader_election.cc:304] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f826241eec6f4799bde3030164f1b95c; no voters:
I20250810 20:28:42.214184 1926 leader_election.cc:290] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:28:42.214839 1931 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:28:42.217252 1931 raft_consensus.cc:695] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 1 LEADER]: Becoming Leader. State: Replica: f826241eec6f4799bde3030164f1b95c, State: Running, Role: LEADER
I20250810 20:28:42.218201 1931 consensus_queue.cc:237] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:28:42.218860 1926 sys_catalog.cc:564] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:28:42.229295 1932 sys_catalog.cc:455] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f826241eec6f4799bde3030164f1b95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } } }
I20250810 20:28:42.231424 1932 sys_catalog.cc:458] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: This master's current role is: LEADER
I20250810 20:28:42.230710 1933 sys_catalog.cc:455] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f826241eec6f4799bde3030164f1b95c. Latest consensus state: current_term: 1 leader_uuid: "f826241eec6f4799bde3030164f1b95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } } }
I20250810 20:28:42.232614 1933 sys_catalog.cc:458] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: This master's current role is: LEADER
I20250810 20:28:42.235878 1940 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:28:42.246819 1940 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:28:42.264555 1940 catalog_manager.cc:1349] Generated new cluster ID: abcb8dce63cb4e04af6103fc6bc754de
I20250810 20:28:42.264894 1940 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:28:42.287793 1940 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 20:28:42.289788 1940 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:28:42.302994 1940 catalog_manager.cc:5955] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Generated new TSK 0
I20250810 20:28:42.304203 1940 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 20:28:42.326295 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:0
--local_ip_for_outbound_sockets=127.30.243.193
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:34401
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250810 20:28:42.650808 1950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:42.651398 1950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:42.652000 1950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:42.685271 1950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:42.686182 1950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:28:42.723241 1950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:34401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:42.724678 1950 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:42.726399 1950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:42.740549 1956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:42.745891 1959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:42.744347 1957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:43.947507 1958 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250810 20:28:43.947590 1950 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:43.952399 1950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:43.956033 1950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:43.957551 1950 hybrid_clock.cc:648] HybridClock initialized: now 1754857723957478 us; error 89 us; skew 500 ppm
I20250810 20:28:43.958698 1950 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:43.966992 1950 webserver.cc:489] Webserver started at http://127.30.243.193:44093/ using document root <none> and password file <none>
I20250810 20:28:43.968375 1950 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:43.968709 1950 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:43.969349 1950 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:43.976317 1950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "8c9d9f82e2b248c99bef8a8a23a22765"
format_stamp: "Formatted at 2025-08-10 20:28:43 on dist-test-slave-7swh"
I20250810 20:28:43.978093 1950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "8c9d9f82e2b248c99bef8a8a23a22765"
format_stamp: "Formatted at 2025-08-10 20:28:43 on dist-test-slave-7swh"
I20250810 20:28:43.988912 1950 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.004s sys 0.004s
I20250810 20:28:43.997040 1966 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:43.998386 1950 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20250810 20:28:43.998852 1950 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8c9d9f82e2b248c99bef8a8a23a22765"
format_stamp: "Formatted at 2025-08-10 20:28:43 on dist-test-slave-7swh"
I20250810 20:28:43.999362 1950 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:44.062564 1950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:44.064606 1950 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:44.065204 1950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:44.068537 1950 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:28:44.074463 1950 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:28:44.074803 1950 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:44.075146 1950 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:28:44.075361 1950 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:44.232364 1950 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:36701
I20250810 20:28:44.232467 2078 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:36701 every 8 connection(s)
I20250810 20:28:44.234997 1950 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 20:28:44.242245 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 1950
I20250810 20:28:44.242666 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 20:28:44.248653 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.194:0
--local_ip_for_outbound_sockets=127.30.243.194
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:34401
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:28:44.257160 2079 heartbeater.cc:344] Connected to a master server at 127.30.243.254:34401
I20250810 20:28:44.257735 2079 heartbeater.cc:461] Registering TS with master...
I20250810 20:28:44.259238 2079 heartbeater.cc:507] Master 127.30.243.254:34401 requested a full tablet report, sending...
I20250810 20:28:44.262595 1891 ts_manager.cc:194] Registered new tserver with Master: 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193:36701)
I20250810 20:28:44.265645 1891 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:48999
W20250810 20:28:44.563930 2083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:44.564399 2083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:44.564851 2083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:44.597528 2083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:44.598364 2083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.194
I20250810 20:28:44.635131 2083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:34401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.194
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:44.636507 2083 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:44.638091 2083 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:44.650691 2089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:45.270892 2079 heartbeater.cc:499] Master 127.30.243.254:34401 was elected leader, sending a full tablet report...
W20250810 20:28:44.652009 2090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:45.896513 2092 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:45.900624 2091 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1247 milliseconds
I20250810 20:28:45.900748 2083 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:45.901996 2083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:45.904865 2083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:45.906345 2083 hybrid_clock.cc:648] HybridClock initialized: now 1754857725906298 us; error 64 us; skew 500 ppm
I20250810 20:28:45.907229 2083 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:45.914368 2083 webserver.cc:489] Webserver started at http://127.30.243.194:33271/ using document root <none> and password file <none>
I20250810 20:28:45.915423 2083 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:45.915645 2083 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:45.916146 2083 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:45.921080 2083 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "d75a9cec40d2474bab109ad0c755329b"
format_stamp: "Formatted at 2025-08-10 20:28:45 on dist-test-slave-7swh"
I20250810 20:28:45.922281 2083 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "d75a9cec40d2474bab109ad0c755329b"
format_stamp: "Formatted at 2025-08-10 20:28:45 on dist-test-slave-7swh"
I20250810 20:28:45.929806 2083 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250810 20:28:45.936343 2099 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:45.937906 2083 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20250810 20:28:45.938269 2083 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d75a9cec40d2474bab109ad0c755329b"
format_stamp: "Formatted at 2025-08-10 20:28:45 on dist-test-slave-7swh"
I20250810 20:28:45.938591 2083 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:46.011076 2083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:46.012580 2083 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:46.013062 2083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:46.015595 2083 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:28:46.020004 2083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:28:46.020224 2083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:46.020478 2083 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:28:46.020632 2083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:46.166661 2083 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.194:39875
I20250810 20:28:46.166771 2211 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.194:39875 every 8 connection(s)
I20250810 20:28:46.169341 2083 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 20:28:46.175006 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 2083
I20250810 20:28:46.175599 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 20:28:46.183784 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.195:0
--local_ip_for_outbound_sockets=127.30.243.195
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:34401
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:28:46.200948 2212 heartbeater.cc:344] Connected to a master server at 127.30.243.254:34401
I20250810 20:28:46.201491 2212 heartbeater.cc:461] Registering TS with master...
I20250810 20:28:46.203168 2212 heartbeater.cc:507] Master 127.30.243.254:34401 requested a full tablet report, sending...
I20250810 20:28:46.205739 1891 ts_manager.cc:194] Registered new tserver with Master: d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875)
I20250810 20:28:46.207826 1891 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.194:44821
W20250810 20:28:46.515630 2216 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:46.516124 2216 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:46.516571 2216 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:46.549464 2216 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:46.550347 2216 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.195
I20250810 20:28:46.588290 2216 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:34401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.195
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:46.589900 2216 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:46.591567 2216 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:46.604867 2222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:47.211951 2212 heartbeater.cc:499] Master 127.30.243.254:34401 was elected leader, sending a full tablet report...
W20250810 20:28:46.605444 2223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:47.897728 2216 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.293s user 0.369s sys 0.912s
W20250810 20:28:47.898221 2216 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.294s user 0.371s sys 0.914s
W20250810 20:28:47.899026 2224 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1293 milliseconds
W20250810 20:28:47.899979 2225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:47.900049 2216 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:47.901407 2216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:47.905957 2216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:47.907382 2216 hybrid_clock.cc:648] HybridClock initialized: now 1754857727907345 us; error 50 us; skew 500 ppm
I20250810 20:28:47.908236 2216 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:47.916184 2216 webserver.cc:489] Webserver started at http://127.30.243.195:40409/ using document root <none> and password file <none>
I20250810 20:28:47.917243 2216 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:47.917488 2216 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:47.917955 2216 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:28:47.922678 2216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "d5eb1279d0c74061ad230b2438a17127"
format_stamp: "Formatted at 2025-08-10 20:28:47 on dist-test-slave-7swh"
I20250810 20:28:47.923930 2216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "d5eb1279d0c74061ad230b2438a17127"
format_stamp: "Formatted at 2025-08-10 20:28:47 on dist-test-slave-7swh"
I20250810 20:28:47.932346 2216 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20250810 20:28:47.938966 2232 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:47.940439 2216 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250810 20:28:47.940860 2216 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d5eb1279d0c74061ad230b2438a17127"
format_stamp: "Formatted at 2025-08-10 20:28:47 on dist-test-slave-7swh"
I20250810 20:28:47.941215 2216 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:48.013967 2216 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:48.015561 2216 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:48.016034 2216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:48.018572 2216 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:28:48.022854 2216 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:28:48.023176 2216 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250810 20:28:48.023444 2216 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:28:48.023605 2216 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:48.168519 2216 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.195:40483
I20250810 20:28:48.168635 2344 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.195:40483 every 8 connection(s)
I20250810 20:28:48.171311 2216 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 20:28:48.172353 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 2216
I20250810 20:28:48.172881 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 20:28:48.198871 2345 heartbeater.cc:344] Connected to a master server at 127.30.243.254:34401
I20250810 20:28:48.199340 2345 heartbeater.cc:461] Registering TS with master...
I20250810 20:28:48.200704 2345 heartbeater.cc:507] Master 127.30.243.254:34401 requested a full tablet report, sending...
I20250810 20:28:48.203033 1891 ts_manager.cc:194] Registered new tserver with Master: d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483)
I20250810 20:28:48.204280 1891 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.195:43179
I20250810 20:28:48.210703 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:28:48.245468 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:28:48.245829 31695 test_util.cc:276] Using random seed: -1484390349
I20250810 20:28:48.289489 1891 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59790:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250810 20:28:48.332543 2014 tablet_service.cc:1468] Processing CreateTablet for tablet 9b40b999e5774f4090201bdc5215328d (DEFAULT_TABLE table=TestTable [id=16d12fbcf82849de9dde115e2e02dd2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:28:48.334005 2014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b40b999e5774f4090201bdc5215328d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:48.356274 2365 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap starting.
I20250810 20:28:48.361737 2365 tablet_bootstrap.cc:654] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:48.363569 2365 log.cc:826] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:48.368304 2365 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: No bootstrap required, opened a new log
I20250810 20:28:48.368768 2365 ts_tablet_manager.cc:1397] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20250810 20:28:48.387283 2365 raft_consensus.cc:357] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } }
I20250810 20:28:48.387887 2365 raft_consensus.cc:383] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:48.388123 2365 raft_consensus.cc:738] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9d9f82e2b248c99bef8a8a23a22765, State: Initialized, Role: FOLLOWER
I20250810 20:28:48.388810 2365 consensus_queue.cc:260] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } }
I20250810 20:28:48.389341 2365 raft_consensus.cc:397] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:28:48.389699 2365 raft_consensus.cc:491] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:28:48.390080 2365 raft_consensus.cc:3058] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:48.394713 2365 raft_consensus.cc:513] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } }
I20250810 20:28:48.395787 2365 leader_election.cc:304] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c9d9f82e2b248c99bef8a8a23a22765; no voters:
I20250810 20:28:48.397559 2365 leader_election.cc:290] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:28:48.397939 2367 raft_consensus.cc:2802] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:28:48.402220 2367 raft_consensus.cc:695] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 1 LEADER]: Becoming Leader. State: Replica: 8c9d9f82e2b248c99bef8a8a23a22765, State: Running, Role: LEADER
I20250810 20:28:48.403412 2367 consensus_queue.cc:237] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } }
I20250810 20:28:48.403683 2365 ts_tablet_manager.cc:1428] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Time spent starting tablet: real 0.035s user 0.027s sys 0.009s
I20250810 20:28:48.416958 1891 catalog_manager.cc:5582] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193). New cstate: current_term: 1 leader_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } health_report { overall_health: HEALTHY } } }
I20250810 20:28:48.607744 31695 test_util.cc:276] Using random seed: -1484028462
I20250810 20:28:48.631464 1885 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59794:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250810 20:28:48.661530 2280 tablet_service.cc:1468] Processing CreateTablet for tablet f3dfb6396c8c4682a541b794033f317c (DEFAULT_TABLE table=TestTable1 [id=19e728efa18f4aed929d56e59989b33f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:28:48.663190 2280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3dfb6396c8c4682a541b794033f317c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:48.684240 2386 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Bootstrap starting.
I20250810 20:28:48.690517 2386 tablet_bootstrap.cc:654] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:48.692353 2386 log.cc:826] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:48.697301 2386 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: No bootstrap required, opened a new log
I20250810 20:28:48.697978 2386 ts_tablet_manager.cc:1397] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20250810 20:28:48.715932 2386 raft_consensus.cc:357] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } }
I20250810 20:28:48.716506 2386 raft_consensus.cc:383] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:48.716699 2386 raft_consensus.cc:738] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5eb1279d0c74061ad230b2438a17127, State: Initialized, Role: FOLLOWER
I20250810 20:28:48.717358 2386 consensus_queue.cc:260] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } }
I20250810 20:28:48.717897 2386 raft_consensus.cc:397] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:28:48.718181 2386 raft_consensus.cc:491] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:28:48.718461 2386 raft_consensus.cc:3058] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:48.722868 2386 raft_consensus.cc:513] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } }
I20250810 20:28:48.723840 2386 leader_election.cc:304] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5eb1279d0c74061ad230b2438a17127; no voters:
I20250810 20:28:48.725589 2386 leader_election.cc:290] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:28:48.726111 2388 raft_consensus.cc:2802] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:28:48.728260 2388 raft_consensus.cc:695] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 1 LEADER]: Becoming Leader. State: Replica: d5eb1279d0c74061ad230b2438a17127, State: Running, Role: LEADER
I20250810 20:28:48.728794 2345 heartbeater.cc:499] Master 127.30.243.254:34401 was elected leader, sending a full tablet report...
I20250810 20:28:48.729192 2388 consensus_queue.cc:237] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } }
I20250810 20:28:48.730087 2386 ts_tablet_manager.cc:1428] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Time spent starting tablet: real 0.032s user 0.016s sys 0.017s
I20250810 20:28:48.738682 1885 catalog_manager.cc:5582] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5eb1279d0c74061ad230b2438a17127 (127.30.243.195). New cstate: current_term: 1 leader_uuid: "d5eb1279d0c74061ad230b2438a17127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } health_report { overall_health: HEALTHY } } }
I20250810 20:28:48.914079 31695 test_util.cc:276] Using random seed: -1483722117
I20250810 20:28:48.939814 1887 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59802:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250810 20:28:48.972780 2147 tablet_service.cc:1468] Processing CreateTablet for tablet c302392458b449cc93b04cc1b61e0c78 (DEFAULT_TABLE table=TestTable2 [id=3459f678b1a54ac1a9a7948a8f7a3558]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:28:48.974553 2147 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c302392458b449cc93b04cc1b61e0c78. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:28:48.995330 2407 tablet_bootstrap.cc:492] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Bootstrap starting.
I20250810 20:28:49.001561 2407 tablet_bootstrap.cc:654] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Neither blocks nor log segments found. Creating new log.
I20250810 20:28:49.003492 2407 log.cc:826] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:49.008517 2407 tablet_bootstrap.cc:492] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: No bootstrap required, opened a new log
I20250810 20:28:49.009078 2407 ts_tablet_manager.cc:1397] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Time spent bootstrapping tablet: real 0.014s user 0.012s sys 0.000s
I20250810 20:28:49.027951 2407 raft_consensus.cc:357] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:28:49.028659 2407 raft_consensus.cc:383] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:28:49.028959 2407 raft_consensus.cc:738] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75a9cec40d2474bab109ad0c755329b, State: Initialized, Role: FOLLOWER
I20250810 20:28:49.029630 2407 consensus_queue.cc:260] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:28:49.030166 2407 raft_consensus.cc:397] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:28:49.030432 2407 raft_consensus.cc:491] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:28:49.030738 2407 raft_consensus.cc:3058] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:28:49.035404 2407 raft_consensus.cc:513] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:28:49.036289 2407 leader_election.cc:304] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d75a9cec40d2474bab109ad0c755329b; no voters:
I20250810 20:28:49.038167 2407 leader_election.cc:290] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:28:49.038535 2409 raft_consensus.cc:2802] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:28:49.041210 2409 raft_consensus.cc:695] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 1 LEADER]: Becoming Leader. State: Replica: d75a9cec40d2474bab109ad0c755329b, State: Running, Role: LEADER
I20250810 20:28:49.042492 2407 ts_tablet_manager.cc:1428] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Time spent starting tablet: real 0.033s user 0.027s sys 0.007s
I20250810 20:28:49.042232 2409 consensus_queue.cc:237] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:28:49.053810 1887 catalog_manager.cc:5582] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b reported cstate change: term changed from 0 to 1, leader changed from <none> to d75a9cec40d2474bab109ad0c755329b (127.30.243.194). New cstate: current_term: 1 leader_uuid: "d75a9cec40d2474bab109ad0c755329b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } health_report { overall_health: HEALTHY } } }
I20250810 20:28:49.337584 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 1858
W20250810 20:28:49.434120 2079 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:34401 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:34401: connect: Connection refused (error 111)
W20250810 20:28:49.754185 2345 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:34401 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:34401: connect: Connection refused (error 111)
W20250810 20:28:50.073607 2212 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:34401 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:34401: connect: Connection refused (error 111)
I20250810 20:28:54.304059 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 1950
I20250810 20:28:54.328557 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 2083
I20250810 20:28:54.356072 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 2216
I20250810 20:28:54.385826 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:34401
--webserver_interface=127.30.243.254
--webserver_port=38047
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:34401 with env {}
W20250810 20:28:54.700548 2486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:54.701243 2486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:54.701668 2486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:54.734045 2486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:28:54.734357 2486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:54.734566 2486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:28:54.734764 2486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:28:54.770879 2486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:34401
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=38047
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:54.772218 2486 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:54.773888 2486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:54.785352 2492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:54.786715 2493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:56.014900 2495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:56.017211 2494 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1227 milliseconds
I20250810 20:28:56.017341 2486 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:56.018672 2486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:56.021522 2486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:56.022888 2486 hybrid_clock.cc:648] HybridClock initialized: now 1754857736022848 us; error 60 us; skew 500 ppm
I20250810 20:28:56.023844 2486 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:56.030471 2486 webserver.cc:489] Webserver started at http://127.30.243.254:38047/ using document root <none> and password file <none>
I20250810 20:28:56.031539 2486 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:56.031773 2486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:56.040122 2486 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.008s sys 0.000s
I20250810 20:28:56.045018 2502 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:56.046298 2486 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250810 20:28:56.046665 2486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f826241eec6f4799bde3030164f1b95c"
format_stamp: "Formatted at 2025-08-10 20:28:41 on dist-test-slave-7swh"
I20250810 20:28:56.048772 2486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:56.121953 2486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:56.123615 2486 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:56.124121 2486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:56.204728 2486 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:34401
I20250810 20:28:56.204816 2553 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:34401 every 8 connection(s)
I20250810 20:28:56.207775 2486 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 20:28:56.216315 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 2486
I20250810 20:28:56.217839 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:36701
--local_ip_for_outbound_sockets=127.30.243.193
--tserver_master_addrs=127.30.243.254:34401
--webserver_port=44093
--webserver_interface=127.30.243.193
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:28:56.219698 2554 sys_catalog.cc:263] Verifying existing consensus state
I20250810 20:28:56.226080 2554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Bootstrap starting.
I20250810 20:28:56.242110 2554 log.cc:826] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:56.305665 2554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:28:56.306834 2554 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Bootstrap complete.
I20250810 20:28:56.341881 2554 raft_consensus.cc:357] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:28:56.344810 2554 raft_consensus.cc:738] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: f826241eec6f4799bde3030164f1b95c, State: Initialized, Role: FOLLOWER
I20250810 20:28:56.345690 2554 consensus_queue.cc:260] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:28:56.346233 2554 raft_consensus.cc:397] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:28:56.346505 2554 raft_consensus.cc:491] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:28:56.346825 2554 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 2 FOLLOWER]: Advancing to term 3
I20250810 20:28:56.353257 2554 raft_consensus.cc:513] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:28:56.354146 2554 leader_election.cc:304] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f826241eec6f4799bde3030164f1b95c; no voters:
I20250810 20:28:56.356411 2554 leader_election.cc:290] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [CANDIDATE]: Term 3 election: Requested vote from peers
I20250810 20:28:56.356815 2558 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 3 FOLLOWER]: Leader election won for term 3
I20250810 20:28:56.360179 2558 raft_consensus.cc:695] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 3 LEADER]: Becoming Leader. State: Replica: f826241eec6f4799bde3030164f1b95c, State: Running, Role: LEADER
I20250810 20:28:56.361097 2558 consensus_queue.cc:237] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:28:56.361682 2554 sys_catalog.cc:564] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:28:56.372841 2560 sys_catalog.cc:455] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f826241eec6f4799bde3030164f1b95c. Latest consensus state: current_term: 3 leader_uuid: "f826241eec6f4799bde3030164f1b95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } } }
I20250810 20:28:56.375172 2560 sys_catalog.cc:458] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: This master's current role is: LEADER
I20250810 20:28:56.374390 2559 sys_catalog.cc:455] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "f826241eec6f4799bde3030164f1b95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } } }
I20250810 20:28:56.376507 2559 sys_catalog.cc:458] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: This master's current role is: LEADER
I20250810 20:28:56.381986 2565 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:28:56.401221 2565 catalog_manager.cc:671] Loaded metadata for table TestTable [id=1b3887b3dc6d4373b52ca9c23550af53]
I20250810 20:28:56.403970 2565 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=3459f678b1a54ac1a9a7948a8f7a3558]
I20250810 20:28:56.405799 2565 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=621bf303be984fd2a4764817137e72d1]
I20250810 20:28:56.414399 2565 tablet_loader.cc:96] loaded metadata for tablet 9b40b999e5774f4090201bdc5215328d (table TestTable [id=1b3887b3dc6d4373b52ca9c23550af53])
I20250810 20:28:56.416128 2565 tablet_loader.cc:96] loaded metadata for tablet c302392458b449cc93b04cc1b61e0c78 (table TestTable2 [id=3459f678b1a54ac1a9a7948a8f7a3558])
I20250810 20:28:56.417615 2565 tablet_loader.cc:96] loaded metadata for tablet f3dfb6396c8c4682a541b794033f317c (table TestTable1 [id=621bf303be984fd2a4764817137e72d1])
I20250810 20:28:56.419293 2565 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:28:56.425380 2565 catalog_manager.cc:1261] Loaded cluster ID: abcb8dce63cb4e04af6103fc6bc754de
I20250810 20:28:56.425777 2565 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:28:56.434849 2565 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:28:56.435213 2576 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250810 20:28:56.440734 2565 catalog_manager.cc:5966] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Loaded TSK: 0
I20250810 20:28:56.442545 2565 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250810 20:28:56.592350 2556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:56.592933 2556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:56.593483 2556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:56.628873 2556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:56.629834 2556 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:28:56.669749 2556 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:36701
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=44093
--tserver_master_addrs=127.30.243.254:34401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:56.671197 2556 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:56.672827 2556 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:56.686831 2582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:56.688249 2583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:58.089885 2581 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 2556
W20250810 20:28:58.215652 2581 kernel_stack_watchdog.cc:198] Thread 2556 stuck at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 20:28:58.223470 2556 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.535s user 0.540s sys 0.984s
W20250810 20:28:58.224016 2556 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.536s user 0.540s sys 0.984s
W20250810 20:28:58.224779 2584 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1535 milliseconds
W20250810 20:28:58.225912 2585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:28:58.226004 2556 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:28:58.227550 2556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:28:58.230487 2556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:28:58.232034 2556 hybrid_clock.cc:648] HybridClock initialized: now 1754857738231937 us; error 102 us; skew 500 ppm
I20250810 20:28:58.233211 2556 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:28:58.241449 2556 webserver.cc:489] Webserver started at http://127.30.243.193:44093/ using document root <none> and password file <none>
I20250810 20:28:58.242838 2556 fs_manager.cc:362] Metadata directory not provided
I20250810 20:28:58.243218 2556 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:28:58.253582 2556 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.005s
I20250810 20:28:58.258877 2592 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:28:58.260353 2556 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250810 20:28:58.260743 2556 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8c9d9f82e2b248c99bef8a8a23a22765"
format_stamp: "Formatted at 2025-08-10 20:28:43 on dist-test-slave-7swh"
I20250810 20:28:58.262809 2556 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:28:58.315276 2556 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:28:58.316813 2556 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:28:58.317289 2556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:28:58.320513 2556 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:28:58.327456 2599 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 20:28:58.339177 2556 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 20:28:58.339459 2556 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.014s user 0.002s sys 0.000s
I20250810 20:28:58.339771 2556 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 20:28:58.344974 2556 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 20:28:58.345209 2556 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250810 20:28:58.345741 2599 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap starting.
I20250810 20:28:58.428725 2599 log.cc:826] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Log is configured to *not* fsync() on all Append() calls
I20250810 20:28:58.538463 2556 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:36701
I20250810 20:28:58.538738 2706 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:36701 every 8 connection(s)
I20250810 20:28:58.542418 2556 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 20:28:58.552711 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 2556
I20250810 20:28:58.554515 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.194:39875
--local_ip_for_outbound_sockets=127.30.243.194
--tserver_master_addrs=127.30.243.254:34401
--webserver_port=33271
--webserver_interface=127.30.243.194
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:28:58.594024 2599 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:28:58.595635 2599 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap complete.
I20250810 20:28:58.598374 2599 ts_tablet_manager.cc:1397] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Time spent bootstrapping tablet: real 0.253s user 0.157s sys 0.088s
I20250810 20:28:58.631477 2707 heartbeater.cc:344] Connected to a master server at 127.30.243.254:34401
I20250810 20:28:58.631989 2707 heartbeater.cc:461] Registering TS with master...
I20250810 20:28:58.633261 2707 heartbeater.cc:507] Master 127.30.243.254:34401 requested a full tablet report, sending...
I20250810 20:28:58.630836 2599 raft_consensus.cc:357] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } }
I20250810 20:28:58.633898 2599 raft_consensus.cc:738] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9d9f82e2b248c99bef8a8a23a22765, State: Initialized, Role: FOLLOWER
I20250810 20:28:58.634881 2599 consensus_queue.cc:260] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } }
I20250810 20:28:58.635556 2599 raft_consensus.cc:397] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:28:58.635959 2599 raft_consensus.cc:491] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:28:58.636507 2599 raft_consensus.cc:3058] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:28:58.644645 2519 ts_manager.cc:194] Registered new tserver with Master: 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193:36701)
I20250810 20:28:58.645095 2599 raft_consensus.cc:513] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } }
I20250810 20:28:58.645936 2599 leader_election.cc:304] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c9d9f82e2b248c99bef8a8a23a22765; no voters:
I20250810 20:28:58.652837 2519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:42889
I20250810 20:28:58.660614 2599 leader_election.cc:290] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250810 20:28:58.661041 2712 raft_consensus.cc:2802] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 20:28:58.666625 2712 raft_consensus.cc:695] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 LEADER]: Becoming Leader. State: Replica: 8c9d9f82e2b248c99bef8a8a23a22765, State: Running, Role: LEADER
I20250810 20:28:58.667729 2712 consensus_queue.cc:237] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } }
I20250810 20:28:58.671051 2707 heartbeater.cc:499] Master 127.30.243.254:34401 was elected leader, sending a full tablet report...
I20250810 20:28:58.673882 2599 ts_tablet_manager.cc:1428] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Time spent starting tablet: real 0.074s user 0.039s sys 0.021s
I20250810 20:28:58.679726 2519 catalog_manager.cc:5582] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: term changed from 0 to 2, leader changed from <none> to 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193), VOTER 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193) added. New cstate: current_term: 2 leader_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } health_report { overall_health: HEALTHY } } }
W20250810 20:28:58.730656 2519 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b40b999e5774f4090201bdc5215328d with cas_config_opid_index -1: no extra replica candidate found for tablet 9b40b999e5774f4090201bdc5215328d (table TestTable [id=1b3887b3dc6d4373b52ca9c23550af53]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250810 20:28:58.962149 2711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:28:58.962730 2711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:28:58.963361 2711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:28:58.996830 2711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:28:58.997749 2711 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.194
I20250810 20:28:59.033689 2711 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.194:39875
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.243.194
--webserver_port=33271
--tserver_master_addrs=127.30.243.254:34401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.194
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:28:59.035178 2711 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:28:59.036861 2711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:28:59.050304 2726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:28:59.052583 2727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:00.422433 2729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:00.425686 2728 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1369 milliseconds
W20250810 20:29:00.426052 2711 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.375s user 0.436s sys 0.897s
W20250810 20:29:00.426442 2711 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.376s user 0.436s sys 0.897s
I20250810 20:29:00.426630 2711 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:00.427773 2711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:00.430190 2711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:00.431727 2711 hybrid_clock.cc:648] HybridClock initialized: now 1754857740431656 us; error 46 us; skew 500 ppm
I20250810 20:29:00.433120 2711 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:00.441510 2711 webserver.cc:489] Webserver started at http://127.30.243.194:33271/ using document root <none> and password file <none>
I20250810 20:29:00.442570 2711 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:00.442840 2711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:00.452646 2711 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.008s sys 0.000s
I20250810 20:29:00.458496 2737 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:00.459863 2711 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20250810 20:29:00.460216 2711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d75a9cec40d2474bab109ad0c755329b"
format_stamp: "Formatted at 2025-08-10 20:28:45 on dist-test-slave-7swh"
I20250810 20:29:00.462615 2711 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:00.539649 2711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:00.541215 2711 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:00.541683 2711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:00.545089 2711 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:29:00.551347 2744 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 20:29:00.559155 2711 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 20:29:00.559434 2711 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250810 20:29:00.559737 2711 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 20:29:00.564795 2711 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 20:29:00.565032 2711 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250810 20:29:00.565874 2744 tablet_bootstrap.cc:492] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Bootstrap starting.
I20250810 20:29:00.642972 2744 log.cc:826] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Log is configured to *not* fsync() on all Append() calls
I20250810 20:29:00.728955 2744 tablet_bootstrap.cc:492] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:00.730049 2744 tablet_bootstrap.cc:492] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Bootstrap complete.
I20250810 20:29:00.731595 2744 ts_tablet_manager.cc:1397] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Time spent bootstrapping tablet: real 0.166s user 0.111s sys 0.052s
I20250810 20:29:00.749431 2744 raft_consensus.cc:357] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:29:00.752996 2744 raft_consensus.cc:738] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75a9cec40d2474bab109ad0c755329b, State: Initialized, Role: FOLLOWER
I20250810 20:29:00.753747 2744 consensus_queue.cc:260] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:29:00.754596 2744 raft_consensus.cc:397] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:29:00.755004 2744 raft_consensus.cc:491] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:29:00.755349 2744 raft_consensus.cc:3058] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:29:00.759979 2711 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.194:39875
I20250810 20:29:00.760119 2851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.194:39875 every 8 connection(s)
I20250810 20:29:00.760918 2744 raft_consensus.cc:513] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:29:00.761579 2744 leader_election.cc:304] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d75a9cec40d2474bab109ad0c755329b; no voters:
I20250810 20:29:00.762539 2711 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 20:29:00.763684 2744 leader_election.cc:290] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [CANDIDATE]: Term 2 election: Requested vote from peers
I20250810 20:29:00.764165 2852 raft_consensus.cc:2802] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Leader election won for term 2
I20250810 20:29:00.770099 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 2711
I20250810 20:29:00.773046 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.195:40483
--local_ip_for_outbound_sockets=127.30.243.195
--tserver_master_addrs=127.30.243.254:34401
--webserver_port=40409
--webserver_interface=127.30.243.195
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:29:00.783133 2852 raft_consensus.cc:695] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 2 LEADER]: Becoming Leader. State: Replica: d75a9cec40d2474bab109ad0c755329b, State: Running, Role: LEADER
I20250810 20:29:00.784775 2852 consensus_queue.cc:237] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:29:00.800436 2853 heartbeater.cc:344] Connected to a master server at 127.30.243.254:34401
I20250810 20:29:00.801034 2853 heartbeater.cc:461] Registering TS with master...
I20250810 20:29:00.802222 2744 ts_tablet_manager.cc:1428] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Time spent starting tablet: real 0.070s user 0.034s sys 0.033s
I20250810 20:29:00.802464 2853 heartbeater.cc:507] Master 127.30.243.254:34401 requested a full tablet report, sending...
I20250810 20:29:00.814339 2519 ts_manager.cc:194] Registered new tserver with Master: d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875)
I20250810 20:29:00.815994 2519 catalog_manager.cc:5582] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "d75a9cec40d2474bab109ad0c755329b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } health_report { overall_health: HEALTHY } } }
I20250810 20:29:00.827190 2519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.194:43355
I20250810 20:29:00.830418 2853 heartbeater.cc:499] Master 127.30.243.254:34401 was elected leader, sending a full tablet report...
I20250810 20:29:01.048466 2662 consensus_queue.cc:237] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } }
I20250810 20:29:01.057080 2869 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER d75a9cec40d2474bab109ad0c755329b (127.30.243.194) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } } }
I20250810 20:29:01.071322 2506 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b40b999e5774f4090201bdc5215328d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250810 20:29:01.074040 2519 catalog_manager.cc:5582] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: config changed from index -1 to 9, NON_VOTER d75a9cec40d2474bab109ad0c755329b (127.30.243.194) added. New cstate: current_term: 2 leader_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250810 20:29:01.087747 2595 consensus_peers.cc:489] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 -> Peer d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875): Couldn't send request to peer d75a9cec40d2474bab109ad0c755329b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b40b999e5774f4090201bdc5215328d. This is attempt 1: this message will repeat every 5th retry.
W20250810 20:29:01.128973 2860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:01.129455 2860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:01.129904 2860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:01.163950 2860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:01.164778 2860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.195
I20250810 20:29:01.203123 2860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.195:40483
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.243.195
--webserver_port=40409
--tserver_master_addrs=127.30.243.254:34401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.195
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:01.204593 2860 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:01.206358 2860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:01.221778 2878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:29:01.613714 2885 ts_tablet_manager.cc:927] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Initiating tablet copy from peer 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193:36701)
I20250810 20:29:01.616358 2885 tablet_copy_client.cc:323] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: tablet copy: Beginning tablet copy session from remote peer at address 127.30.243.193:36701
I20250810 20:29:01.645576 2682 tablet_copy_service.cc:140] P 8c9d9f82e2b248c99bef8a8a23a22765: Received BeginTabletCopySession request for tablet 9b40b999e5774f4090201bdc5215328d from peer d75a9cec40d2474bab109ad0c755329b ({username='slave'} at 127.30.243.194:57533)
I20250810 20:29:01.646111 2682 tablet_copy_service.cc:161] P 8c9d9f82e2b248c99bef8a8a23a22765: Beginning new tablet copy session on tablet 9b40b999e5774f4090201bdc5215328d from peer d75a9cec40d2474bab109ad0c755329b at {username='slave'} at 127.30.243.194:57533: session id = d75a9cec40d2474bab109ad0c755329b-9b40b999e5774f4090201bdc5215328d
I20250810 20:29:01.650926 2682 tablet_copy_source_session.cc:215] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Tablet Copy: opened 0 blocks and 1 log segments
I20250810 20:29:01.654364 2885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b40b999e5774f4090201bdc5215328d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:29:01.668208 2885 tablet_copy_client.cc:806] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: tablet copy: Starting download of 0 data blocks...
I20250810 20:29:01.668951 2885 tablet_copy_client.cc:670] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: tablet copy: Starting download of 1 WAL segments...
I20250810 20:29:01.672686 2885 tablet_copy_client.cc:538] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250810 20:29:01.678984 2885 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Bootstrap starting.
I20250810 20:29:01.771381 2885 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:01.772275 2885 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Bootstrap complete.
I20250810 20:29:01.772882 2885 ts_tablet_manager.cc:1397] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Time spent bootstrapping tablet: real 0.094s user 0.087s sys 0.010s
I20250810 20:29:01.775364 2885 raft_consensus.cc:357] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } }
I20250810 20:29:01.776014 2885 raft_consensus.cc:738] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d75a9cec40d2474bab109ad0c755329b, State: Initialized, Role: LEARNER
I20250810 20:29:01.776602 2885 consensus_queue.cc:260] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } }
I20250810 20:29:01.799714 2885 ts_tablet_manager.cc:1428] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Time spent starting tablet: real 0.027s user 0.005s sys 0.015s
I20250810 20:29:01.807926 2682 tablet_copy_service.cc:342] P 8c9d9f82e2b248c99bef8a8a23a22765: Request end of tablet copy session d75a9cec40d2474bab109ad0c755329b-9b40b999e5774f4090201bdc5215328d received from {username='slave'} at 127.30.243.194:57533
I20250810 20:29:01.808415 2682 tablet_copy_service.cc:434] P 8c9d9f82e2b248c99bef8a8a23a22765: ending tablet copy session d75a9cec40d2474bab109ad0c755329b-9b40b999e5774f4090201bdc5215328d on tablet 9b40b999e5774f4090201bdc5215328d with peer d75a9cec40d2474bab109ad0c755329b
I20250810 20:29:02.015038 2807 raft_consensus.cc:1215] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250810 20:29:02.456913 2891 raft_consensus.cc:1062] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: attempting to promote NON_VOTER d75a9cec40d2474bab109ad0c755329b to VOTER
I20250810 20:29:02.458806 2891 consensus_queue.cc:237] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:02.465451 2807 raft_consensus.cc:1273] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 LEARNER]: Refusing update from remote peer 8c9d9f82e2b248c99bef8a8a23a22765: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250810 20:29:02.467200 2891 consensus_queue.cc:1035] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250810 20:29:02.477152 2892 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, d75a9cec40d2474bab109ad0c755329b (127.30.243.194) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } }
I20250810 20:29:02.483791 2807 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, d75a9cec40d2474bab109ad0c755329b (127.30.243.194) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } }
I20250810 20:29:02.496423 2519 catalog_manager.cc:5582] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: config changed from index 9 to 10, d75a9cec40d2474bab109ad0c755329b (127.30.243.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250810 20:29:01.222633 2879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:01.226779 2881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:02.615480 2880 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1391 milliseconds
I20250810 20:29:02.615604 2860 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:02.616775 2860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:02.619589 2860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:02.621062 2860 hybrid_clock.cc:648] HybridClock initialized: now 1754857742621015 us; error 69 us; skew 500 ppm
I20250810 20:29:02.621927 2860 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:02.629114 2860 webserver.cc:489] Webserver started at http://127.30.243.195:40409/ using document root <none> and password file <none>
I20250810 20:29:02.630092 2860 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:02.630327 2860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:02.638907 2860 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.004s
I20250810 20:29:02.644634 2904 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:02.645823 2860 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250810 20:29:02.646145 2860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d5eb1279d0c74061ad230b2438a17127"
format_stamp: "Formatted at 2025-08-10 20:28:47 on dist-test-slave-7swh"
I20250810 20:29:02.648208 2860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:02.705380 2860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:02.706883 2860 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:02.707335 2860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:02.709841 2860 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:29:02.715934 2911 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 20:29:02.724408 2860 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 20:29:02.724702 2860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250810 20:29:02.725023 2860 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 20:29:02.730086 2860 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 20:29:02.730310 2860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250810 20:29:02.730759 2911 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Bootstrap starting.
I20250810 20:29:02.799292 2911 log.cc:826] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Log is configured to *not* fsync() on all Append() calls
I20250810 20:29:02.891299 2911 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:02.892191 2911 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Bootstrap complete.
I20250810 20:29:02.894085 2911 ts_tablet_manager.cc:1397] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Time spent bootstrapping tablet: real 0.164s user 0.111s sys 0.048s
I20250810 20:29:02.911027 2911 raft_consensus.cc:357] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } }
I20250810 20:29:02.913674 2911 raft_consensus.cc:738] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5eb1279d0c74061ad230b2438a17127, State: Initialized, Role: FOLLOWER
I20250810 20:29:02.914870 2911 consensus_queue.cc:260] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } }
I20250810 20:29:02.915707 2911 raft_consensus.cc:397] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:29:02.915998 2911 raft_consensus.cc:491] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:29:02.916360 2911 raft_consensus.cc:3058] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:29:02.922981 2911 raft_consensus.cc:513] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } }
I20250810 20:29:02.924261 2911 leader_election.cc:304] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d5eb1279d0c74061ad230b2438a17127; no voters:
I20250810 20:29:02.927737 2911 leader_election.cc:290] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250810 20:29:02.928102 3011 raft_consensus.cc:2802] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 20:29:02.932467 2911 ts_tablet_manager.cc:1428] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Time spent starting tablet: real 0.038s user 0.034s sys 0.005s
I20250810 20:29:02.933487 3011 raft_consensus.cc:695] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 LEADER]: Becoming Leader. State: Replica: d5eb1279d0c74061ad230b2438a17127, State: Running, Role: LEADER
I20250810 20:29:02.934630 3011 consensus_queue.cc:237] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } }
I20250810 20:29:02.947316 2860 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.195:40483
I20250810 20:29:02.947875 3023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.195:40483 every 8 connection(s)
I20250810 20:29:02.949898 2860 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 20:29:02.957249 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 2860
I20250810 20:29:02.975791 3024 heartbeater.cc:344] Connected to a master server at 127.30.243.254:34401
I20250810 20:29:02.976357 3024 heartbeater.cc:461] Registering TS with master...
I20250810 20:29:02.977674 3024 heartbeater.cc:507] Master 127.30.243.254:34401 requested a full tablet report, sending...
I20250810 20:29:02.981216 2519 ts_manager.cc:194] Registered new tserver with Master: d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483)
I20250810 20:29:02.982275 2519 catalog_manager.cc:5582] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 reported cstate change: term changed from 0 to 2, leader changed from <none> to d5eb1279d0c74061ad230b2438a17127 (127.30.243.195), VOTER d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) added. New cstate: current_term: 2 leader_uuid: "d5eb1279d0c74061ad230b2438a17127" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } health_report { overall_health: HEALTHY } } }
I20250810 20:29:02.991888 2519 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.195:54957
I20250810 20:29:02.993912 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:29:02.996980 3024 heartbeater.cc:499] Master 127.30.243.254:34401 was elected leader, sending a full tablet report...
I20250810 20:29:02.999843 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250810 20:29:03.004370 31695 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250810 20:29:03.008584 2974 consensus_queue.cc:237] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: NON_VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: true } }
I20250810 20:29:03.012197 3014 raft_consensus.cc:2953] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: NON_VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: true } } }
I20250810 20:29:03.021366 2503 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f3dfb6396c8c4682a541b794033f317c with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250810 20:29:03.023577 2908 consensus_peers.cc:489] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 -> Peer 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193:36701): Couldn't send request to peer 8c9d9f82e2b248c99bef8a8a23a22765. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3dfb6396c8c4682a541b794033f317c. This is attempt 1: this message will repeat every 5th retry.
I20250810 20:29:03.024099 2518 catalog_manager.cc:5582] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 reported cstate change: config changed from index -1 to 8, NON_VOTER 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193) added. New cstate: current_term: 2 leader_uuid: "d5eb1279d0c74061ad230b2438a17127" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: NON_VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250810 20:29:03.033272 2974 consensus_queue.cc:237] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: NON_VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: true } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } }
W20250810 20:29:03.038241 2908 consensus_peers.cc:489] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 -> Peer 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193:36701): Couldn't send request to peer 8c9d9f82e2b248c99bef8a8a23a22765. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3dfb6396c8c4682a541b794033f317c. This is attempt 1: this message will repeat every 5th retry.
I20250810 20:29:03.036650 3014 raft_consensus.cc:2953] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER d75a9cec40d2474bab109ad0c755329b (127.30.243.194) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: NON_VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: true } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } } }
I20250810 20:29:03.047343 2503 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f3dfb6396c8c4682a541b794033f317c with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250810 20:29:03.048731 2907 consensus_peers.cc:489] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 -> Peer d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875): Couldn't send request to peer d75a9cec40d2474bab109ad0c755329b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f3dfb6396c8c4682a541b794033f317c. This is attempt 1: this message will repeat every 5th retry.
I20250810 20:29:03.051239 2518 catalog_manager.cc:5582] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 reported cstate change: config changed from index 8 to 9, NON_VOTER d75a9cec40d2474bab109ad0c755329b (127.30.243.194) added. New cstate: current_term: 2 leader_uuid: "d5eb1279d0c74061ad230b2438a17127" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: NON_VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250810 20:29:03.129699 2662 consensus_queue.cc:237] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: true } }
I20250810 20:29:03.134920 2807 raft_consensus.cc:1273] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Refusing update from remote peer 8c9d9f82e2b248c99bef8a8a23a22765: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250810 20:29:03.136200 2892 consensus_queue.cc:1035] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250810 20:29:03.142545 2891 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: true } } }
I20250810 20:29:03.144079 2807 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: true } } }
W20250810 20:29:03.144891 2593 consensus_peers.cc:489] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 -> Peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483): Couldn't send request to peer d5eb1279d0c74061ad230b2438a17127. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b40b999e5774f4090201bdc5215328d. This is attempt 1: this message will repeat every 5th retry.
I20250810 20:29:03.150578 2506 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b40b999e5774f4090201bdc5215328d with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 6)
I20250810 20:29:03.153600 2518 catalog_manager.cc:5582] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: config changed from index 10 to 11, NON_VOTER d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) added. New cstate: current_term: 2 leader_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250810 20:29:03.302317 2505 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9b40b999e5774f4090201bdc5215328d with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250810 20:29:03.576102 3037 ts_tablet_manager.cc:927] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Initiating tablet copy from peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483)
I20250810 20:29:03.577569 3037 tablet_copy_client.cc:323] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: tablet copy: Beginning tablet copy session from remote peer at address 127.30.243.195:40483
I20250810 20:29:03.579301 2994 tablet_copy_service.cc:140] P d5eb1279d0c74061ad230b2438a17127: Received BeginTabletCopySession request for tablet f3dfb6396c8c4682a541b794033f317c from peer 8c9d9f82e2b248c99bef8a8a23a22765 ({username='slave'} at 127.30.243.193:60157)
I20250810 20:29:03.579788 2994 tablet_copy_service.cc:161] P d5eb1279d0c74061ad230b2438a17127: Beginning new tablet copy session on tablet f3dfb6396c8c4682a541b794033f317c from peer 8c9d9f82e2b248c99bef8a8a23a22765 at {username='slave'} at 127.30.243.193:60157: session id = 8c9d9f82e2b248c99bef8a8a23a22765-f3dfb6396c8c4682a541b794033f317c
I20250810 20:29:03.584585 2994 tablet_copy_source_session.cc:215] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Tablet Copy: opened 0 blocks and 1 log segments
I20250810 20:29:03.588104 3037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3dfb6396c8c4682a541b794033f317c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:29:03.599897 3037 tablet_copy_client.cc:806] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: tablet copy: Starting download of 0 data blocks...
I20250810 20:29:03.600510 3037 tablet_copy_client.cc:670] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: tablet copy: Starting download of 1 WAL segments...
I20250810 20:29:03.605327 3037 tablet_copy_client.cc:538] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250810 20:29:03.611231 3037 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap starting.
I20250810 20:29:03.662083 3040 ts_tablet_manager.cc:927] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Initiating tablet copy from peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483)
I20250810 20:29:03.663589 3040 tablet_copy_client.cc:323] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: tablet copy: Beginning tablet copy session from remote peer at address 127.30.243.195:40483
I20250810 20:29:03.675856 2994 tablet_copy_service.cc:140] P d5eb1279d0c74061ad230b2438a17127: Received BeginTabletCopySession request for tablet f3dfb6396c8c4682a541b794033f317c from peer d75a9cec40d2474bab109ad0c755329b ({username='slave'} at 127.30.243.194:44285)
I20250810 20:29:03.676318 2994 tablet_copy_service.cc:161] P d5eb1279d0c74061ad230b2438a17127: Beginning new tablet copy session on tablet f3dfb6396c8c4682a541b794033f317c from peer d75a9cec40d2474bab109ad0c755329b at {username='slave'} at 127.30.243.194:44285: session id = d75a9cec40d2474bab109ad0c755329b-f3dfb6396c8c4682a541b794033f317c
I20250810 20:29:03.680500 2994 tablet_copy_source_session.cc:215] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Tablet Copy: opened 0 blocks and 1 log segments
I20250810 20:29:03.682776 3040 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3dfb6396c8c4682a541b794033f317c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:29:03.693035 3040 tablet_copy_client.cc:806] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: tablet copy: Starting download of 0 data blocks...
I20250810 20:29:03.693594 3040 tablet_copy_client.cc:670] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: tablet copy: Starting download of 1 WAL segments...
I20250810 20:29:03.693706 3037 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:03.694474 3037 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap complete.
I20250810 20:29:03.695124 3037 ts_tablet_manager.cc:1397] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Time spent bootstrapping tablet: real 0.084s user 0.071s sys 0.012s
I20250810 20:29:03.696837 3037 raft_consensus.cc:357] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: NON_VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: true } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } }
I20250810 20:29:03.697003 3040 tablet_copy_client.cc:538] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250810 20:29:03.697327 3037 raft_consensus.cc:738] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 8c9d9f82e2b248c99bef8a8a23a22765, State: Initialized, Role: LEARNER
I20250810 20:29:03.697754 3037 consensus_queue.cc:260] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: NON_VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: true } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } }
I20250810 20:29:03.699321 3037 ts_tablet_manager.cc:1428] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250810 20:29:03.700944 2994 tablet_copy_service.cc:342] P d5eb1279d0c74061ad230b2438a17127: Request end of tablet copy session 8c9d9f82e2b248c99bef8a8a23a22765-f3dfb6396c8c4682a541b794033f317c received from {username='slave'} at 127.30.243.193:60157
I20250810 20:29:03.701390 2994 tablet_copy_service.cc:434] P d5eb1279d0c74061ad230b2438a17127: ending tablet copy session 8c9d9f82e2b248c99bef8a8a23a22765-f3dfb6396c8c4682a541b794033f317c on tablet f3dfb6396c8c4682a541b794033f317c with peer 8c9d9f82e2b248c99bef8a8a23a22765
I20250810 20:29:03.706674 3040 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Bootstrap starting.
I20250810 20:29:03.716770 3044 ts_tablet_manager.cc:927] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Initiating tablet copy from peer 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193:36701)
I20250810 20:29:03.718747 3044 tablet_copy_client.cc:323] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: tablet copy: Beginning tablet copy session from remote peer at address 127.30.243.193:36701
I20250810 20:29:03.720849 2682 tablet_copy_service.cc:140] P 8c9d9f82e2b248c99bef8a8a23a22765: Received BeginTabletCopySession request for tablet 9b40b999e5774f4090201bdc5215328d from peer d5eb1279d0c74061ad230b2438a17127 ({username='slave'} at 127.30.243.195:57939)
I20250810 20:29:03.721491 2682 tablet_copy_service.cc:161] P 8c9d9f82e2b248c99bef8a8a23a22765: Beginning new tablet copy session on tablet 9b40b999e5774f4090201bdc5215328d from peer d5eb1279d0c74061ad230b2438a17127 at {username='slave'} at 127.30.243.195:57939: session id = d5eb1279d0c74061ad230b2438a17127-9b40b999e5774f4090201bdc5215328d
I20250810 20:29:03.725996 2682 tablet_copy_source_session.cc:215] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Tablet Copy: opened 0 blocks and 1 log segments
I20250810 20:29:03.729218 3044 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b40b999e5774f4090201bdc5215328d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:29:03.744273 3044 tablet_copy_client.cc:806] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: tablet copy: Starting download of 0 data blocks...
I20250810 20:29:03.744910 3044 tablet_copy_client.cc:670] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: tablet copy: Starting download of 1 WAL segments...
I20250810 20:29:03.749574 3044 tablet_copy_client.cc:538] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250810 20:29:03.755903 3044 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Bootstrap starting.
I20250810 20:29:03.811271 3040 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:03.812060 3040 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Bootstrap complete.
I20250810 20:29:03.812645 3040 ts_tablet_manager.cc:1397] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Time spent bootstrapping tablet: real 0.106s user 0.099s sys 0.008s
I20250810 20:29:03.814714 3040 raft_consensus.cc:357] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: NON_VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: true } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } }
I20250810 20:29:03.815527 3040 raft_consensus.cc:738] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d75a9cec40d2474bab109ad0c755329b, State: Initialized, Role: LEARNER
I20250810 20:29:03.816080 3040 consensus_queue.cc:260] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: NON_VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: true } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } }
I20250810 20:29:03.819499 3040 ts_tablet_manager.cc:1428] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20250810 20:29:03.821285 2994 tablet_copy_service.cc:342] P d5eb1279d0c74061ad230b2438a17127: Request end of tablet copy session d75a9cec40d2474bab109ad0c755329b-f3dfb6396c8c4682a541b794033f317c received from {username='slave'} at 127.30.243.194:44285
I20250810 20:29:03.821708 2994 tablet_copy_service.cc:434] P d5eb1279d0c74061ad230b2438a17127: ending tablet copy session d75a9cec40d2474bab109ad0c755329b-f3dfb6396c8c4682a541b794033f317c on tablet f3dfb6396c8c4682a541b794033f317c with peer d75a9cec40d2474bab109ad0c755329b
I20250810 20:29:03.853832 3044 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:03.854457 3044 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Bootstrap complete.
I20250810 20:29:03.854899 3044 ts_tablet_manager.cc:1397] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Time spent bootstrapping tablet: real 0.099s user 0.071s sys 0.024s
I20250810 20:29:03.857036 3044 raft_consensus.cc:357] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: true } }
I20250810 20:29:03.857527 3044 raft_consensus.cc:738] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: d5eb1279d0c74061ad230b2438a17127, State: Initialized, Role: LEARNER
I20250810 20:29:03.857918 3044 consensus_queue.cc:260] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: NON_VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: true } }
I20250810 20:29:03.859685 3044 ts_tablet_manager.cc:1428] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250810 20:29:03.861915 2682 tablet_copy_service.cc:342] P 8c9d9f82e2b248c99bef8a8a23a22765: Request end of tablet copy session d5eb1279d0c74061ad230b2438a17127-9b40b999e5774f4090201bdc5215328d received from {username='slave'} at 127.30.243.195:57939
I20250810 20:29:03.862269 2682 tablet_copy_service.cc:434] P 8c9d9f82e2b248c99bef8a8a23a22765: ending tablet copy session d5eb1279d0c74061ad230b2438a17127-9b40b999e5774f4090201bdc5215328d on tablet 9b40b999e5774f4090201bdc5215328d with peer d5eb1279d0c74061ad230b2438a17127
I20250810 20:29:04.009717 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8c9d9f82e2b248c99bef8a8a23a22765 to finish bootstrapping
I20250810 20:29:04.025054 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d75a9cec40d2474bab109ad0c755329b to finish bootstrapping
I20250810 20:29:04.038730 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver d5eb1279d0c74061ad230b2438a17127 to finish bootstrapping
I20250810 20:29:04.146538 2662 raft_consensus.cc:1215] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250810 20:29:04.167335 2807 raft_consensus.cc:1215] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250810 20:29:04.184742 2974 raft_consensus.cc:1215] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250810 20:29:04.342293 2641 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 20:29:04.344544 2954 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 20:29:04.353346 2786 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 20:29:04.531984 3060 raft_consensus.cc:1062] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: attempting to promote NON_VOTER 8c9d9f82e2b248c99bef8a8a23a22765 to VOTER
I20250810 20:29:04.534304 3060 consensus_queue.cc:237] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } }
I20250810 20:29:04.543555 2807 raft_consensus.cc:1273] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 LEARNER]: Refusing update from remote peer d5eb1279d0c74061ad230b2438a17127: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250810 20:29:04.545101 2662 raft_consensus.cc:1273] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 LEARNER]: Refusing update from remote peer d5eb1279d0c74061ad230b2438a17127: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250810 20:29:04.545269 3060 consensus_queue.cc:1035] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250810 20:29:04.546756 3036 consensus_queue.cc:1035] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250810 20:29:04.562522 3059 raft_consensus.cc:1025] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 LEADER]: attempt to promote peer d75a9cec40d2474bab109ad0c755329b: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20250810 20:29:04.566200 3036 raft_consensus.cc:2953] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } } }
I20250810 20:29:04.568207 2662 raft_consensus.cc:2953] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } } }
I20250810 20:29:04.578249 2807 raft_consensus.cc:2953] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } } }
I20250810 20:29:04.581034 2519 catalog_manager.cc:5582] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 reported cstate change: config changed from index 9 to 10, 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d5eb1279d0c74061ad230b2438a17127" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: NON_VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: true } health_report { overall_health: HEALTHY } } }
I20250810 20:29:04.590070 3036 raft_consensus.cc:1062] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: attempting to promote NON_VOTER d75a9cec40d2474bab109ad0c755329b to VOTER
I20250810 20:29:04.592937 3036 consensus_queue.cc:237] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:04.601104 2807 raft_consensus.cc:1273] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 LEARNER]: Refusing update from remote peer d5eb1279d0c74061ad230b2438a17127: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250810 20:29:04.603389 2661 raft_consensus.cc:1273] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Refusing update from remote peer d5eb1279d0c74061ad230b2438a17127: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250810 20:29:04.605327 3060 consensus_queue.cc:1035] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
f826241eec6f4799bde3030164f1b95c | 127.30.243.254:34401 | HEALTHY
Unusual flags for Master:
I20250810 20:29:04.611223 3036 consensus_queue.cc:1035] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.30.243.212:43713 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
I20250810 20:29:04.614185 3059 raft_consensus.cc:2953] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d75a9cec40d2474bab109ad0c755329b (127.30.243.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } }
I20250810 20:29:04.616426 2806 raft_consensus.cc:2953] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d75a9cec40d2474bab109ad0c755329b (127.30.243.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } }
I20250810 20:29:04.618764 2661 raft_consensus.cc:2953] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, d75a9cec40d2474bab109ad0c755329b (127.30.243.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } }
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
8c9d9f82e2b248c99bef8a8a23a22765 | 127.30.243.193:36701 | HEALTHY | <none> | 1 | 0
d5eb1279d0c74061ad230b2438a17127 | 127.30.243.195:40483 | HEALTHY | <none> | 1 | 0
d75a9cec40d2474bab109ad0c755329b | 127.30.243.194:39875 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.30.243.193 | experimental | 127.30.243.193:36701
local_ip_for_outbound_sockets | 127.30.243.194 | experimental | 127.30.243.194:39875
local_ip_for_outbound_sockets | 127.30.243.195 | experimental | 127.30.243.195:40483
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.30.243.193:36701
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.30.243.194:39875
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.30.243.195:40483
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.30.243.212:43713 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250810 20:29:04.641729 2517 catalog_manager.cc:5582] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b reported cstate change: config changed from index 10 to 11, d75a9cec40d2474bab109ad0c755329b (127.30.243.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "d5eb1279d0c74061ad230b2438a17127" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } }
I20250810 20:29:04.645525 31695 log_verifier.cc:126] Checking tablet 9b40b999e5774f4090201bdc5215328d
I20250810 20:29:04.740614 3062 raft_consensus.cc:1062] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: attempting to promote NON_VOTER d5eb1279d0c74061ad230b2438a17127 to VOTER
I20250810 20:29:04.742146 3062 consensus_queue.cc:237] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
I20250810 20:29:04.747896 2974 raft_consensus.cc:1273] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [term 2 LEARNER]: Refusing update from remote peer 8c9d9f82e2b248c99bef8a8a23a22765: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250810 20:29:04.749215 2806 raft_consensus.cc:1273] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Refusing update from remote peer 8c9d9f82e2b248c99bef8a8a23a22765: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250810 20:29:04.749323 3062 consensus_queue.cc:1035] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250810 20:29:04.750702 3089 consensus_queue.cc:1035] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250810 20:29:04.756150 31695 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 9b40b999e5774f4090201bdc5215328d
I20250810 20:29:04.756618 31695 log_verifier.cc:126] Checking tablet c302392458b449cc93b04cc1b61e0c78
I20250810 20:29:04.773367 3062 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } } }
I20250810 20:29:04.775709 2974 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } } }
I20250810 20:29:04.777449 2806 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } } }
I20250810 20:29:04.790627 2519 catalog_manager.cc:5582] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: config changed from index 11 to 12, d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250810 20:29:04.801713 31695 log_verifier.cc:177] Verified matching terms for 8 ops in tablet c302392458b449cc93b04cc1b61e0c78
I20250810 20:29:04.802019 31695 log_verifier.cc:126] Checking tablet f3dfb6396c8c4682a541b794033f317c
I20250810 20:29:04.905771 31695 log_verifier.cc:177] Verified matching terms for 11 ops in tablet f3dfb6396c8c4682a541b794033f317c
I20250810 20:29:04.906193 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 2486
I20250810 20:29:04.932648 31695 minidump.cc:252] Setting minidump size limit to 20M
I20250810 20:29:04.934067 31695 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:04.935556 31695 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:04.955866 3095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:04.957042 3096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:04.963291 3098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:29:05.094408 31695 server_base.cc:1047] running on GCE node
I20250810 20:29:05.096030 31695 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250810 20:29:05.096386 31695 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250810 20:29:05.096554 31695 hybrid_clock.cc:648] HybridClock initialized: now 1754857745096537 us; error 0 us; skew 500 ppm
I20250810 20:29:05.097646 31695 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:05.101255 31695 webserver.cc:489] Webserver started at http://0.0.0.0:38287/ using document root <none> and password file <none>
I20250810 20:29:05.102210 31695 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:05.102420 31695 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:05.109727 31695 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250810 20:29:05.114878 3103 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:05.116274 31695 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250810 20:29:05.116683 31695 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f826241eec6f4799bde3030164f1b95c"
format_stamp: "Formatted at 2025-08-10 20:28:41 on dist-test-slave-7swh"
I20250810 20:29:05.118697 31695 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:05.135404 31695 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:05.137100 31695 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:05.137586 31695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:05.147488 31695 sys_catalog.cc:263] Verifying existing consensus state
W20250810 20:29:05.151546 31695 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.30.243.254:34401 exists but no master address supplied!
I20250810 20:29:05.154490 31695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Bootstrap starting.
I20250810 20:29:05.200940 31695 log.cc:826] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Log is configured to *not* fsync() on all Append() calls
W20250810 20:29:05.231554 3024 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:34401 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:34401: connect: Connection refused (error 111)
I20250810 20:29:05.269134 31695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:05.269906 31695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Bootstrap complete.
I20250810 20:29:05.284245 31695 raft_consensus.cc:357] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:29:05.284896 31695 raft_consensus.cc:738] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: f826241eec6f4799bde3030164f1b95c, State: Initialized, Role: FOLLOWER
I20250810 20:29:05.285705 31695 consensus_queue.cc:260] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:29:05.286222 31695 raft_consensus.cc:397] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:29:05.286484 31695 raft_consensus.cc:491] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:29:05.286830 31695 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 3 FOLLOWER]: Advancing to term 4
I20250810 20:29:05.292804 31695 raft_consensus.cc:513] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:29:05.293648 31695 leader_election.cc:304] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f826241eec6f4799bde3030164f1b95c; no voters:
I20250810 20:29:05.295116 31695 leader_election.cc:290] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [CANDIDATE]: Term 4 election: Requested vote from peers
I20250810 20:29:05.295488 3113 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 4 FOLLOWER]: Leader election won for term 4
I20250810 20:29:05.296964 3113 raft_consensus.cc:695] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 4 LEADER]: Becoming Leader. State: Replica: f826241eec6f4799bde3030164f1b95c, State: Running, Role: LEADER
I20250810 20:29:05.297813 3113 consensus_queue.cc:237] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:29:05.305075 3114 sys_catalog.cc:455] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "f826241eec6f4799bde3030164f1b95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } } }
I20250810 20:29:05.305946 3114 sys_catalog.cc:458] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: This master's current role is: LEADER
I20250810 20:29:05.306428 3115 sys_catalog.cc:455] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f826241eec6f4799bde3030164f1b95c. Latest consensus state: current_term: 4 leader_uuid: "f826241eec6f4799bde3030164f1b95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } } }
I20250810 20:29:05.307042 3115 sys_catalog.cc:458] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: This master's current role is: LEADER
I20250810 20:29:05.335438 31695 tablet_replica.cc:331] stopping tablet replica
I20250810 20:29:05.336212 31695 raft_consensus.cc:2241] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 4 LEADER]: Raft consensus shutting down.
I20250810 20:29:05.336827 31695 raft_consensus.cc:2270] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 4 FOLLOWER]: Raft consensus is shut down!
I20250810 20:29:05.340077 31695 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250810 20:29:05.340747 31695 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250810 20:29:05.371819 31695 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250810 20:29:05.814189 2853 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:34401 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:34401: connect: Connection refused (error 111)
W20250810 20:29:05.822948 2707 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:34401 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:34401: connect: Connection refused (error 111)
I20250810 20:29:10.620163 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 2556
I20250810 20:29:10.648651 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 2711
I20250810 20:29:10.678598 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 2860
I20250810 20:29:10.710161 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:34401
--webserver_interface=127.30.243.254
--webserver_port=38047
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:34401 with env {}
W20250810 20:29:11.028545 3186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:11.029179 3186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:11.029600 3186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:11.062419 3186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:29:11.062745 3186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:11.062985 3186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:29:11.063202 3186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:29:11.101477 3186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:34401
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:34401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=38047
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:11.102777 3186 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:11.104365 3186 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:11.116236 3192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:11.117832 3193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:11.120404 3195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:12.343407 3194 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1223 milliseconds
I20250810 20:29:12.343554 3186 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:12.344861 3186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:12.347599 3186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:12.348975 3186 hybrid_clock.cc:648] HybridClock initialized: now 1754857752348930 us; error 67 us; skew 500 ppm
I20250810 20:29:12.349803 3186 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:12.357241 3186 webserver.cc:489] Webserver started at http://127.30.243.254:38047/ using document root <none> and password file <none>
I20250810 20:29:12.358268 3186 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:12.358489 3186 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:12.366621 3186 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250810 20:29:12.371472 3202 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:12.372920 3186 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 20:29:12.373302 3186 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f826241eec6f4799bde3030164f1b95c"
format_stamp: "Formatted at 2025-08-10 20:28:41 on dist-test-slave-7swh"
I20250810 20:29:12.375355 3186 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:12.441009 3186 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:12.442569 3186 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:12.443116 3186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:12.518597 3186 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:34401
I20250810 20:29:12.518688 3253 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:34401 every 8 connection(s)
I20250810 20:29:12.521538 3186 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 20:29:12.526427 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 3186
I20250810 20:29:12.527963 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:36701
--local_ip_for_outbound_sockets=127.30.243.193
--tserver_master_addrs=127.30.243.254:34401
--webserver_port=44093
--webserver_interface=127.30.243.193
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:29:12.532771 3254 sys_catalog.cc:263] Verifying existing consensus state
I20250810 20:29:12.537757 3254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Bootstrap starting.
I20250810 20:29:12.548607 3254 log.cc:826] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Log is configured to *not* fsync() on all Append() calls
I20250810 20:29:12.630831 3254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:12.631683 3254 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Bootstrap complete.
I20250810 20:29:12.650980 3254 raft_consensus.cc:357] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:29:12.653175 3254 raft_consensus.cc:738] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: f826241eec6f4799bde3030164f1b95c, State: Initialized, Role: FOLLOWER
I20250810 20:29:12.654044 3254 consensus_queue.cc:260] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:29:12.654559 3254 raft_consensus.cc:397] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:29:12.654820 3254 raft_consensus.cc:491] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:29:12.655153 3254 raft_consensus.cc:3058] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 5 FOLLOWER]: Advancing to term 6
I20250810 20:29:12.660463 3254 raft_consensus.cc:513] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:29:12.661182 3254 leader_election.cc:304] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f826241eec6f4799bde3030164f1b95c; no voters:
I20250810 20:29:12.663452 3254 leader_election.cc:290] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [CANDIDATE]: Term 6 election: Requested vote from peers
I20250810 20:29:12.663872 3258 raft_consensus.cc:2802] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 6 FOLLOWER]: Leader election won for term 6
I20250810 20:29:12.667268 3258 raft_consensus.cc:695] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [term 6 LEADER]: Becoming Leader. State: Replica: f826241eec6f4799bde3030164f1b95c, State: Running, Role: LEADER
I20250810 20:29:12.668165 3258 consensus_queue.cc:237] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } }
I20250810 20:29:12.668787 3254 sys_catalog.cc:564] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:29:12.680545 3259 sys_catalog.cc:455] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "f826241eec6f4799bde3030164f1b95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } } }
I20250810 20:29:12.682628 3259 sys_catalog.cc:458] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: This master's current role is: LEADER
I20250810 20:29:12.681790 3260 sys_catalog.cc:455] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: SysCatalogTable state changed. Reason: New leader f826241eec6f4799bde3030164f1b95c. Latest consensus state: current_term: 6 leader_uuid: "f826241eec6f4799bde3030164f1b95c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f826241eec6f4799bde3030164f1b95c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 34401 } } }
I20250810 20:29:12.684074 3260 sys_catalog.cc:458] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c [sys.catalog]: This master's current role is: LEADER
I20250810 20:29:12.689589 3266 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:29:12.707387 3266 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=3459f678b1a54ac1a9a7948a8f7a3558]
I20250810 20:29:12.709995 3266 catalog_manager.cc:671] Loaded metadata for table TestTable [id=5f22469ace0947b298f281652a27902b]
I20250810 20:29:12.712255 3266 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=621bf303be984fd2a4764817137e72d1]
I20250810 20:29:12.724227 3266 tablet_loader.cc:96] loaded metadata for tablet 9b40b999e5774f4090201bdc5215328d (table TestTable [id=5f22469ace0947b298f281652a27902b])
I20250810 20:29:12.726318 3266 tablet_loader.cc:96] loaded metadata for tablet c302392458b449cc93b04cc1b61e0c78 (table TestTable2 [id=3459f678b1a54ac1a9a7948a8f7a3558])
I20250810 20:29:12.728272 3266 tablet_loader.cc:96] loaded metadata for tablet f3dfb6396c8c4682a541b794033f317c (table TestTable1 [id=621bf303be984fd2a4764817137e72d1])
I20250810 20:29:12.730667 3266 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:29:12.738802 3266 catalog_manager.cc:1261] Loaded cluster ID: abcb8dce63cb4e04af6103fc6bc754de
I20250810 20:29:12.739404 3266 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:29:12.752225 3266 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:29:12.760800 3266 catalog_manager.cc:5966] T 00000000000000000000000000000000 P f826241eec6f4799bde3030164f1b95c: Loaded TSK: 0
I20250810 20:29:12.764148 3266 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250810 20:29:12.907507 3256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:12.908056 3256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:12.908576 3256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:12.943343 3256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:12.944258 3256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:29:12.981967 3256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:36701
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=44093
--tserver_master_addrs=127.30.243.254:34401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:12.983403 3256 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:12.985073 3256 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:12.999540 3281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:13.003149 3284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:13.002004 3282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:14.195766 3283 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250810 20:29:14.195813 3256 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:14.201159 3256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:14.204805 3256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:14.206360 3256 hybrid_clock.cc:648] HybridClock initialized: now 1754857754206323 us; error 38 us; skew 500 ppm
I20250810 20:29:14.207532 3256 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:14.215662 3256 webserver.cc:489] Webserver started at http://127.30.243.193:44093/ using document root <none> and password file <none>
I20250810 20:29:14.217038 3256 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:14.217356 3256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:14.228825 3256 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.005s sys 0.003s
I20250810 20:29:14.235630 3291 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:14.237000 3256 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250810 20:29:14.237412 3256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "8c9d9f82e2b248c99bef8a8a23a22765"
format_stamp: "Formatted at 2025-08-10 20:28:43 on dist-test-slave-7swh"
I20250810 20:29:14.240381 3256 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:14.309098 3256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:14.310738 3256 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:14.311228 3256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:14.313916 3256 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:29:14.321035 3298 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250810 20:29:14.340247 3256 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250810 20:29:14.340553 3256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.021s user 0.002s sys 0.000s
I20250810 20:29:14.340864 3256 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250810 20:29:14.346710 3298 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap starting.
I20250810 20:29:14.350023 3256 ts_tablet_manager.cc:610] Registered 2 tablets
I20250810 20:29:14.350255 3256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.009s user 0.007s sys 0.000s
I20250810 20:29:14.418702 3298 log.cc:826] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Log is configured to *not* fsync() on all Append() calls
I20250810 20:29:14.551497 3298 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:14.553073 3298 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap complete.
I20250810 20:29:14.555536 3298 ts_tablet_manager.cc:1397] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Time spent bootstrapping tablet: real 0.209s user 0.152s sys 0.050s
I20250810 20:29:14.583783 3298 raft_consensus.cc:357] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:14.587895 3298 raft_consensus.cc:738] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9d9f82e2b248c99bef8a8a23a22765, State: Initialized, Role: FOLLOWER
I20250810 20:29:14.589205 3298 consensus_queue.cc:260] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:14.590780 3256 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:36701
I20250810 20:29:14.590912 3405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:36701 every 8 connection(s)
I20250810 20:29:14.593542 3298 ts_tablet_manager.cc:1428] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765: Time spent starting tablet: real 0.038s user 0.029s sys 0.006s
I20250810 20:29:14.594794 3298 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap starting.
I20250810 20:29:14.594861 3256 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 20:29:14.603423 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 3256
I20250810 20:29:14.605562 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.194:39875
--local_ip_for_outbound_sockets=127.30.243.194
--tserver_master_addrs=127.30.243.254:34401
--webserver_port=33271
--webserver_interface=127.30.243.194
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:29:14.634920 3407 heartbeater.cc:344] Connected to a master server at 127.30.243.254:34401
I20250810 20:29:14.635509 3407 heartbeater.cc:461] Registering TS with master...
I20250810 20:29:14.636956 3407 heartbeater.cc:507] Master 127.30.243.254:34401 requested a full tablet report, sending...
I20250810 20:29:14.643841 3219 ts_manager.cc:194] Registered new tserver with Master: 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193:36701)
I20250810 20:29:14.649425 3219 catalog_manager.cc:5582] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: config changed from index -1 to 12, term changed from 0 to 2, VOTER 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193) added, VOTER d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) added, VOTER d75a9cec40d2474bab109ad0c755329b (127.30.243.194) added. New cstate: current_term: 2 committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } } }
I20250810 20:29:14.736321 3219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:39399
I20250810 20:29:14.761260 3298 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:14.762032 3298 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Bootstrap complete.
I20250810 20:29:14.763356 3298 ts_tablet_manager.cc:1397] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Time spent bootstrapping tablet: real 0.169s user 0.137s sys 0.027s
I20250810 20:29:14.765022 3298 raft_consensus.cc:357] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
I20250810 20:29:14.765542 3298 raft_consensus.cc:738] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c9d9f82e2b248c99bef8a8a23a22765, State: Initialized, Role: FOLLOWER
I20250810 20:29:14.766114 3298 consensus_queue.cc:260] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
I20250810 20:29:14.767132 3407 heartbeater.cc:499] Master 127.30.243.254:34401 was elected leader, sending a full tablet report...
I20250810 20:29:14.767776 3298 ts_tablet_manager.cc:1428] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250810 20:29:15.000823 3411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:15.001371 3411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:15.001884 3411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:15.034300 3411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:15.035262 3411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.194
I20250810 20:29:15.071022 3411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.194:39875
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.243.194
--webserver_port=33271
--tserver_master_addrs=127.30.243.254:34401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.194
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:15.072381 3411 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:15.074162 3411 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:15.086071 3423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:29:16.076416 3429 raft_consensus.cc:491] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:29:16.077742 3429 raft_consensus.cc:513] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
W20250810 20:29:16.090101 3292 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.243.195:40483: connect: Connection refused (error 111)
I20250810 20:29:16.092643 3429 leader_election.cc:290] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483), d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875)
W20250810 20:29:16.114529 3292 leader_election.cc:336] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483): Network error: Client connection negotiation failed: client connection to 127.30.243.195:40483: connect: Connection refused (error 111)
W20250810 20:29:16.117887 3294 leader_election.cc:336] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875): Network error: Client connection negotiation failed: client connection to 127.30.243.194:39875: connect: Connection refused (error 111)
I20250810 20:29:16.118871 3294 leader_election.cc:304] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c9d9f82e2b248c99bef8a8a23a22765; no voters: d5eb1279d0c74061ad230b2438a17127, d75a9cec40d2474bab109ad0c755329b
I20250810 20:29:16.120460 3429 raft_consensus.cc:2747] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250810 20:29:16.235821 3429 raft_consensus.cc:491] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:29:16.236367 3429 raft_consensus.cc:513] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
W20250810 20:29:16.249110 3292 leader_election.cc:336] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483): Network error: Client connection negotiation failed: client connection to 127.30.243.195:40483: connect: Connection refused (error 111)
W20250810 20:29:16.249948 3294 leader_election.cc:336] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875): Network error: Client connection negotiation failed: client connection to 127.30.243.194:39875: connect: Connection refused (error 111)
I20250810 20:29:16.250424 3294 leader_election.cc:304] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c9d9f82e2b248c99bef8a8a23a22765; no voters: d5eb1279d0c74061ad230b2438a17127, d75a9cec40d2474bab109ad0c755329b
I20250810 20:29:16.252238 3429 leader_election.cc:290] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875), d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483)
I20250810 20:29:16.252760 3429 raft_consensus.cc:2747] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250810 20:29:15.088001 3424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:16.407451 3426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:16.409970 3411 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.323s user 0.493s sys 0.821s
W20250810 20:29:16.410279 3411 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.323s user 0.493s sys 0.821s
W20250810 20:29:16.410188 3425 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1317 milliseconds
I20250810 20:29:16.410533 3411 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:16.411737 3411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:16.414191 3411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:16.415576 3411 hybrid_clock.cc:648] HybridClock initialized: now 1754857756415538 us; error 53 us; skew 500 ppm
I20250810 20:29:16.416474 3411 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:16.429631 3411 webserver.cc:489] Webserver started at http://127.30.243.194:33271/ using document root <none> and password file <none>
I20250810 20:29:16.430670 3411 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:16.430959 3411 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:16.445717 3411 fs_manager.cc:714] Time spent opening directory manager: real 0.011s user 0.006s sys 0.007s
I20250810 20:29:16.453948 3438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:16.455708 3411 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.004s sys 0.002s
I20250810 20:29:16.456265 3411 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "d75a9cec40d2474bab109ad0c755329b"
format_stamp: "Formatted at 2025-08-10 20:28:45 on dist-test-slave-7swh"
I20250810 20:29:16.459584 3411 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:16.561223 3411 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:16.562804 3411 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:16.563323 3411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:16.565948 3411 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:29:16.572095 3445 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250810 20:29:16.589002 3411 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250810 20:29:16.589366 3411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.019s user 0.000s sys 0.002s
I20250810 20:29:16.589722 3411 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250810 20:29:16.598542 3445 tablet_bootstrap.cc:492] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Bootstrap starting.
I20250810 20:29:16.609676 3411 ts_tablet_manager.cc:610] Registered 3 tablets
I20250810 20:29:16.609968 3411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.020s user 0.018s sys 0.000s
I20250810 20:29:16.654222 3445 log.cc:826] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Log is configured to *not* fsync() on all Append() calls
I20250810 20:29:16.785871 3445 tablet_bootstrap.cc:492] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:16.787115 3445 tablet_bootstrap.cc:492] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Bootstrap complete.
I20250810 20:29:16.788249 3411 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.194:39875
I20250810 20:29:16.788399 3552 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.194:39875 every 8 connection(s)
I20250810 20:29:16.789347 3445 ts_tablet_manager.cc:1397] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Time spent bootstrapping tablet: real 0.191s user 0.161s sys 0.025s
I20250810 20:29:16.791963 3411 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 20:29:16.800766 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 3411
I20250810 20:29:16.802764 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.195:40483
--local_ip_for_outbound_sockets=127.30.243.195
--tserver_master_addrs=127.30.243.254:34401
--webserver_port=40409
--webserver_interface=127.30.243.195
--builtin_ntp_servers=127.30.243.212:43713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250810 20:29:16.812741 3445 raft_consensus.cc:357] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:29:16.817895 3445 raft_consensus.cc:738] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75a9cec40d2474bab109ad0c755329b, State: Initialized, Role: FOLLOWER
I20250810 20:29:16.818727 3445 consensus_queue.cc:260] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:29:16.819469 3445 raft_consensus.cc:397] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:29:16.819895 3445 raft_consensus.cc:491] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:29:16.820288 3445 raft_consensus.cc:3058] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Advancing to term 3
I20250810 20:29:16.831056 3445 raft_consensus.cc:513] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:29:16.832314 3445 leader_election.cc:304] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d75a9cec40d2474bab109ad0c755329b; no voters:
I20250810 20:29:16.846334 3445 leader_election.cc:290] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [CANDIDATE]: Term 3 election: Requested vote from peers
I20250810 20:29:16.846769 3558 raft_consensus.cc:2802] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 3 FOLLOWER]: Leader election won for term 3
I20250810 20:29:16.850054 3553 heartbeater.cc:344] Connected to a master server at 127.30.243.254:34401
I20250810 20:29:16.850121 3445 ts_tablet_manager.cc:1428] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b: Time spent starting tablet: real 0.060s user 0.056s sys 0.004s
I20250810 20:29:16.850703 3553 heartbeater.cc:461] Registering TS with master...
I20250810 20:29:16.850997 3445 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Bootstrap starting.
I20250810 20:29:16.852499 3553 heartbeater.cc:507] Master 127.30.243.254:34401 requested a full tablet report, sending...
I20250810 20:29:16.854205 3558 raft_consensus.cc:695] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [term 3 LEADER]: Becoming Leader. State: Replica: d75a9cec40d2474bab109ad0c755329b, State: Running, Role: LEADER
I20250810 20:29:16.855470 3558 consensus_queue.cc:237] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } }
I20250810 20:29:16.867630 3219 ts_manager.cc:194] Registered new tserver with Master: d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875)
I20250810 20:29:16.870034 3219 catalog_manager.cc:5582] T c302392458b449cc93b04cc1b61e0c78 P d75a9cec40d2474bab109ad0c755329b reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "d75a9cec40d2474bab109ad0c755329b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } health_report { overall_health: HEALTHY } } }
I20250810 20:29:16.908185 3219 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.194:40793
I20250810 20:29:16.913189 3553 heartbeater.cc:499] Master 127.30.243.254:34401 was elected leader, sending a full tablet report...
I20250810 20:29:16.975494 3445 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:16.976217 3445 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Bootstrap complete.
I20250810 20:29:16.977555 3445 ts_tablet_manager.cc:1397] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Time spent bootstrapping tablet: real 0.127s user 0.110s sys 0.008s
I20250810 20:29:16.979383 3445 raft_consensus.cc:357] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:16.979895 3445 raft_consensus.cc:738] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75a9cec40d2474bab109ad0c755329b, State: Initialized, Role: FOLLOWER
I20250810 20:29:16.980479 3445 consensus_queue.cc:260] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:16.982048 3445 ts_tablet_manager.cc:1428] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250810 20:29:16.982703 3445 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Bootstrap starting.
I20250810 20:29:17.086135 3445 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:17.086984 3445 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Bootstrap complete.
I20250810 20:29:17.088431 3445 ts_tablet_manager.cc:1397] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Time spent bootstrapping tablet: real 0.106s user 0.092s sys 0.012s
I20250810 20:29:17.090049 3445 raft_consensus.cc:357] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
I20250810 20:29:17.090535 3445 raft_consensus.cc:738] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d75a9cec40d2474bab109ad0c755329b, State: Initialized, Role: FOLLOWER
I20250810 20:29:17.091120 3445 consensus_queue.cc:260] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
I20250810 20:29:17.092793 3445 ts_tablet_manager.cc:1428] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250810 20:29:17.208477 3557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:17.209053 3557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:17.209645 3557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:17.243780 3557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:17.244683 3557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.195
I20250810 20:29:17.280517 3557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:43713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.195:40483
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.243.195
--webserver_port=40409
--tserver_master_addrs=127.30.243.254:34401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.195
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:17.281889 3557 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:17.283592 3557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:17.296582 3574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:29:18.203508 3580 raft_consensus.cc:491] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:29:18.204203 3580 raft_consensus.cc:513] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:18.206885 3580 leader_election.cc:290] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483), d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875)
W20250810 20:29:18.238364 3292 leader_election.cc:336] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483): Network error: Client connection negotiation failed: client connection to 127.30.243.195:40483: connect: Connection refused (error 111)
I20250810 20:29:18.242628 3508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3dfb6396c8c4682a541b794033f317c" candidate_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "d75a9cec40d2474bab109ad0c755329b" is_pre_election: true
I20250810 20:29:18.244401 3508 raft_consensus.cc:2466] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9d9f82e2b248c99bef8a8a23a22765 in term 2.
I20250810 20:29:18.247011 3294 leader_election.cc:304] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c9d9f82e2b248c99bef8a8a23a22765, d75a9cec40d2474bab109ad0c755329b; no voters: d5eb1279d0c74061ad230b2438a17127
I20250810 20:29:18.248875 3580 raft_consensus.cc:2802] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250810 20:29:18.249374 3580 raft_consensus.cc:491] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 20:29:18.249794 3580 raft_consensus.cc:3058] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Advancing to term 3
I20250810 20:29:18.264341 3580 raft_consensus.cc:513] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:18.268625 3508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f3dfb6396c8c4682a541b794033f317c" candidate_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "d75a9cec40d2474bab109ad0c755329b"
I20250810 20:29:18.269620 3508 raft_consensus.cc:3058] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Advancing to term 3
W20250810 20:29:18.274891 3292 leader_election.cc:336] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483): Network error: Client connection negotiation failed: client connection to 127.30.243.195:40483: connect: Connection refused (error 111)
I20250810 20:29:18.275751 3580 leader_election.cc:290] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 election: Requested vote from peers d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483), d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875)
I20250810 20:29:18.282176 3508 raft_consensus.cc:2466] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9d9f82e2b248c99bef8a8a23a22765 in term 3.
I20250810 20:29:18.284065 3294 leader_election.cc:304] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c9d9f82e2b248c99bef8a8a23a22765, d75a9cec40d2474bab109ad0c755329b; no voters: d5eb1279d0c74061ad230b2438a17127
I20250810 20:29:18.285408 3580 raft_consensus.cc:2802] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 3 FOLLOWER]: Leader election won for term 3
I20250810 20:29:18.288255 3580 raft_consensus.cc:695] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [term 3 LEADER]: Becoming Leader. State: Replica: 8c9d9f82e2b248c99bef8a8a23a22765, State: Running, Role: LEADER
I20250810 20:29:18.290269 3580 consensus_queue.cc:237] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:18.312178 3218 catalog_manager.cc:5582] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: term changed from 2 to 3, leader changed from d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) to 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193). New cstate: current_term: 3 leader_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250810 20:29:18.488291 3580 raft_consensus.cc:491] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:29:18.488994 3580 raft_consensus.cc:513] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
I20250810 20:29:18.492509 3508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b40b999e5774f4090201bdc5215328d" candidate_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "d75a9cec40d2474bab109ad0c755329b" is_pre_election: true
I20250810 20:29:18.493324 3508 raft_consensus.cc:2466] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c9d9f82e2b248c99bef8a8a23a22765 in term 2.
I20250810 20:29:18.494773 3294 leader_election.cc:304] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c9d9f82e2b248c99bef8a8a23a22765, d75a9cec40d2474bab109ad0c755329b; no voters:
I20250810 20:29:18.495059 3580 leader_election.cc:290] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875), d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483)
I20250810 20:29:18.495704 3580 raft_consensus.cc:2802] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250810 20:29:18.496068 3580 raft_consensus.cc:491] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 20:29:18.496670 3580 raft_consensus.cc:3058] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 2 FOLLOWER]: Advancing to term 3
W20250810 20:29:18.498824 3292 leader_election.cc:336] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483): Network error: Client connection negotiation failed: client connection to 127.30.243.195:40483: connect: Connection refused (error 111)
I20250810 20:29:18.506342 3580 raft_consensus.cc:513] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
I20250810 20:29:18.509467 3580 leader_election.cc:290] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 election: Requested vote from peers d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875), d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483)
I20250810 20:29:18.510704 3508 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b40b999e5774f4090201bdc5215328d" candidate_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "d75a9cec40d2474bab109ad0c755329b"
I20250810 20:29:18.511380 3508 raft_consensus.cc:3058] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 2 FOLLOWER]: Advancing to term 3
W20250810 20:29:18.517048 3292 leader_election.cc:336] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483): Network error: Client connection negotiation failed: client connection to 127.30.243.195:40483: connect: Connection refused (error 111)
I20250810 20:29:18.520052 3508 raft_consensus.cc:2466] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c9d9f82e2b248c99bef8a8a23a22765 in term 3.
I20250810 20:29:18.521404 3294 leader_election.cc:304] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c9d9f82e2b248c99bef8a8a23a22765, d75a9cec40d2474bab109ad0c755329b; no voters: d5eb1279d0c74061ad230b2438a17127
I20250810 20:29:18.522596 3580 raft_consensus.cc:2802] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 3 FOLLOWER]: Leader election won for term 3
I20250810 20:29:18.524078 3580 raft_consensus.cc:695] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 3 LEADER]: Becoming Leader. State: Replica: 8c9d9f82e2b248c99bef8a8a23a22765, State: Running, Role: LEADER
I20250810 20:29:18.524942 3580 consensus_queue.cc:237] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
I20250810 20:29:18.544080 3218 catalog_manager.cc:5582] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: term changed from 2 to 3, leader changed from <none> to 8c9d9f82e2b248c99bef8a8a23a22765 (127.30.243.193). New cstate: current_term: 3 leader_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250810 20:29:17.297569 3575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:17.300897 3577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:18.601280 3576 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1303 milliseconds
I20250810 20:29:18.601403 3557 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:18.602599 3557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:18.604876 3557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:18.606292 3557 hybrid_clock.cc:648] HybridClock initialized: now 1754857758606244 us; error 61 us; skew 500 ppm
I20250810 20:29:18.607195 3557 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:18.613626 3557 webserver.cc:489] Webserver started at http://127.30.243.195:40409/ using document root <none> and password file <none>
I20250810 20:29:18.614594 3557 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:18.614832 3557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:18.622761 3557 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250810 20:29:18.627710 3596 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:18.628842 3557 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250810 20:29:18.629186 3557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "d5eb1279d0c74061ad230b2438a17127"
format_stamp: "Formatted at 2025-08-10 20:28:47 on dist-test-slave-7swh"
I20250810 20:29:18.631230 3557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:18.682163 3557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:18.683713 3557 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:18.684186 3557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:18.687150 3557 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
W20250810 20:29:18.690490 3292 consensus_peers.cc:489] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 -> Peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483): Couldn't send request to peer d5eb1279d0c74061ad230b2438a17127. Status: Network error: Client connection negotiation failed: client connection to 127.30.243.195:40483: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250810 20:29:18.694183 3603 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250810 20:29:18.707934 3557 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250810 20:29:18.708370 3557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.002s sys 0.001s
I20250810 20:29:18.708763 3557 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250810 20:29:18.715680 3603 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Bootstrap starting.
I20250810 20:29:18.718755 3557 ts_tablet_manager.cc:610] Registered 2 tablets
I20250810 20:29:18.719033 3557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.007s sys 0.004s
I20250810 20:29:18.784631 3603 log.cc:826] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Log is configured to *not* fsync() on all Append() calls
I20250810 20:29:18.811378 3508 raft_consensus.cc:1273] T f3dfb6396c8c4682a541b794033f317c P d75a9cec40d2474bab109ad0c755329b [term 3 FOLLOWER]: Refusing update from remote peer 8c9d9f82e2b248c99bef8a8a23a22765: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250810 20:29:18.814170 3580 consensus_queue.cc:1035] T f3dfb6396c8c4682a541b794033f317c P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250810 20:29:18.911345 3603 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:18.912251 3603 tablet_bootstrap.cc:492] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Bootstrap complete.
I20250810 20:29:18.913784 3603 ts_tablet_manager.cc:1397] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Time spent bootstrapping tablet: real 0.199s user 0.135s sys 0.050s
I20250810 20:29:18.928426 3603 raft_consensus.cc:357] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:18.930852 3603 raft_consensus.cc:738] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5eb1279d0c74061ad230b2438a17127, State: Initialized, Role: FOLLOWER
I20250810 20:29:18.931778 3603 consensus_queue.cc:260] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } } peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } attrs { promote: false } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:18.936949 3603 ts_tablet_manager.cc:1428] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127: Time spent starting tablet: real 0.023s user 0.021s sys 0.003s
I20250810 20:29:18.937759 3603 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Bootstrap starting.
I20250810 20:29:18.949420 3507 raft_consensus.cc:1273] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 3 FOLLOWER]: Refusing update from remote peer 8c9d9f82e2b248c99bef8a8a23a22765: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250810 20:29:18.952119 3585 consensus_queue.cc:1035] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
W20250810 20:29:18.972247 3292 consensus_peers.cc:489] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 -> Peer d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483): Couldn't send request to peer d5eb1279d0c74061ad230b2438a17127. Status: Network error: Client connection negotiation failed: client connection to 127.30.243.195:40483: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250810 20:29:19.011574 3361 consensus_queue.cc:237] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } }
I20250810 20:29:19.018404 3507 raft_consensus.cc:1273] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 3 FOLLOWER]: Refusing update from remote peer 8c9d9f82e2b248c99bef8a8a23a22765: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250810 20:29:19.020663 3557 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.195:40483
I20250810 20:29:19.020263 3703 consensus_queue.cc:1035] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20250810 20:29:19.024415 3557 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 20:29:19.033300 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 3557
I20250810 20:29:19.029541 3703 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } }
I20250810 20:29:19.032361 3507 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } }
I20250810 20:29:19.054193 3720 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.195:40483 every 8 connection(s)
I20250810 20:29:19.080175 3206 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9b40b999e5774f4090201bdc5215328d with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250810 20:29:19.081682 3218 catalog_manager.cc:5582] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: config changed from index 12 to 14, VOTER d5eb1279d0c74061ad230b2438a17127 (127.30.243.195) evicted. New cstate: current_term: 3 leader_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250810 20:29:19.134824 3218 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 9b40b999e5774f4090201bdc5215328d on TS d5eb1279d0c74061ad230b2438a17127: Not found: failed to reset TS proxy: Could not find TS for UUID d5eb1279d0c74061ad230b2438a17127
I20250810 20:29:19.136286 3721 heartbeater.cc:344] Connected to a master server at 127.30.243.254:34401
I20250810 20:29:19.136792 3721 heartbeater.cc:461] Registering TS with master...
I20250810 20:29:19.138073 3721 heartbeater.cc:507] Master 127.30.243.254:34401 requested a full tablet report, sending...
I20250810 20:29:19.140295 3361 consensus_queue.cc:237] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 12, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } }
I20250810 20:29:19.143024 3585 raft_consensus.cc:2953] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER d75a9cec40d2474bab109ad0c755329b (127.30.243.194) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } }
I20250810 20:29:19.143915 3218 ts_manager.cc:194] Registered new tserver with Master: d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483)
I20250810 20:29:19.149012 3218 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.195:37045
I20250810 20:29:19.150702 3206 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 9b40b999e5774f4090201bdc5215328d with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250810 20:29:19.155495 3218 catalog_manager.cc:5582] T 9b40b999e5774f4090201bdc5215328d P 8c9d9f82e2b248c99bef8a8a23a22765 reported cstate change: config changed from index 14 to 15, VOTER d75a9cec40d2474bab109ad0c755329b (127.30.243.194) evicted. New cstate: current_term: 3 leader_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } health_report { overall_health: HEALTHY } } }
I20250810 20:29:19.157234 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:29:19.160248 3603 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:29:19.161260 3603 tablet_bootstrap.cc:492] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Bootstrap complete.
I20250810 20:29:19.162720 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:29:19.162982 3603 ts_tablet_manager.cc:1397] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Time spent bootstrapping tablet: real 0.225s user 0.163s sys 0.025s
I20250810 20:29:19.165153 3603 raft_consensus.cc:357] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
I20250810 20:29:19.166051 3603 raft_consensus.cc:738] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5eb1279d0c74061ad230b2438a17127, State: Initialized, Role: FOLLOWER
I20250810 20:29:19.166757 3603 consensus_queue.cc:260] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "8c9d9f82e2b248c99bef8a8a23a22765" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 36701 } } peers { permanent_uuid: "d75a9cec40d2474bab109ad0c755329b" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 39875 } attrs { promote: false } } peers { permanent_uuid: "d5eb1279d0c74061ad230b2438a17127" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 40483 } attrs { promote: false } }
I20250810 20:29:19.168802 3603 ts_tablet_manager.cc:1428] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250810 20:29:19.169575 3721 heartbeater.cc:499] Master 127.30.243.254:34401 was elected leader, sending a full tablet report...
W20250810 20:29:19.169842 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250810 20:29:19.177376 3488 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b40b999e5774f4090201bdc5215328d with delete_type TABLET_DATA_TOMBSTONED (TS d75a9cec40d2474bab109ad0c755329b not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:53060
I20250810 20:29:19.179894 3731 tablet_replica.cc:331] stopping tablet replica
I20250810 20:29:19.180590 3731 raft_consensus.cc:2241] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 3 FOLLOWER]: Raft consensus shutting down.
I20250810 20:29:19.181105 3731 raft_consensus.cc:2270] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b [term 3 FOLLOWER]: Raft consensus is shut down!
I20250810 20:29:19.184026 3731 ts_tablet_manager.cc:1905] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250810 20:29:19.196530 3731 ts_tablet_manager.cc:1918] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20250810 20:29:19.196905 3731 log.cc:1199] T 9b40b999e5774f4090201bdc5215328d P d75a9cec40d2474bab109ad0c755329b: Deleting WAL directory at /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/wals/9b40b999e5774f4090201bdc5215328d
I20250810 20:29:19.198319 3205 catalog_manager.cc:4928] TS d75a9cec40d2474bab109ad0c755329b (127.30.243.194:39875): tablet 9b40b999e5774f4090201bdc5215328d (table TestTable [id=5f22469ace0947b298f281652a27902b]) successfully deleted
I20250810 20:29:19.200450 3649 tablet_service.cc:1515] Processing DeleteTablet for tablet 9b40b999e5774f4090201bdc5215328d with delete_type TABLET_DATA_TOMBSTONED (TS d5eb1279d0c74061ad230b2438a17127 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:55718
I20250810 20:29:19.206442 3733 tablet_replica.cc:331] stopping tablet replica
I20250810 20:29:19.207192 3733 raft_consensus.cc:2241] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250810 20:29:19.207612 3733 raft_consensus.cc:2270] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250810 20:29:19.210211 3733 ts_tablet_manager.cc:1905] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250810 20:29:19.223129 3733 ts_tablet_manager.cc:1918] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250810 20:29:19.224222 3733 log.cc:1199] T 9b40b999e5774f4090201bdc5215328d P d5eb1279d0c74061ad230b2438a17127: Deleting WAL directory at /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b40b999e5774f4090201bdc5215328d
I20250810 20:29:19.225998 3203 catalog_manager.cc:4928] TS d5eb1279d0c74061ad230b2438a17127 (127.30.243.195:40483): tablet 9b40b999e5774f4090201bdc5215328d (table TestTable [id=5f22469ace0947b298f281652a27902b]) successfully deleted
I20250810 20:29:19.231745 3670 raft_consensus.cc:3058] T f3dfb6396c8c4682a541b794033f317c P d5eb1279d0c74061ad230b2438a17127 [term 2 FOLLOWER]: Advancing to term 3
W20250810 20:29:20.174820 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:21.179340 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:22.182963 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:23.186703 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:24.190631 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:25.194644 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:26.199074 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:27.202595 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:28.206308 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:29.209983 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:30.214233 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:31.219072 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:32.222925 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:33.226672 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:34.230537 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:35.234122 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:36.238746 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:37.242393 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250810 20:29:38.246290 31695 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 9b40b999e5774f4090201bdc5215328d: tablet_id: "9b40b999e5774f4090201bdc5215328d" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250810 20:29:39.249579 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 3256
I20250810 20:29:39.279536 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 3411
I20250810 20:29:39.311681 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 3557
I20250810 20:29:39.335629 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 3186
2025-08-10T20:29:39Z chronyd exiting
I20250810 20:29:39.392237 31695 test_util.cc:183] -----------------------------------------------
I20250810 20:29:39.392463 31695 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1754857648852986-31695-0
[ FAILED ] AdminCliTest.TestRebuildTables (59704 ms)
[----------] 5 tests from AdminCliTest (130469 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250810 20:29:39.396440 31695 test_util.cc:276] Using random seed: -1433239748
I20250810 20:29:39.400823 31695 ts_itest-base.cc:115] Starting cluster with:
I20250810 20:29:39.401005 31695 ts_itest-base.cc:116] --------------
I20250810 20:29:39.401141 31695 ts_itest-base.cc:117] 5 tablet servers
I20250810 20:29:39.401254 31695 ts_itest-base.cc:118] 3 replicas per TS
I20250810 20:29:39.401376 31695 ts_itest-base.cc:119] --------------
2025-08-10T20:29:39Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T20:29:39Z Disabled control of system clock
I20250810 20:29:39.450498 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:38889
--webserver_interface=127.30.243.254
--webserver_port=0
--builtin_ntp_servers=127.30.243.212:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:38889
--raft_prepare_replacement_before_eviction=true with env {}
W20250810 20:29:39.771045 3756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:39.771718 3756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:39.772240 3756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:39.806641 3756 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 20:29:39.807109 3756 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:29:39.807370 3756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:39.807608 3756 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:29:39.807849 3756 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:29:39.844599 3756 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:38889
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:38889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:39.845937 3756 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:39.847659 3756 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:39.860481 3762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:39.860961 3763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:41.263804 3761 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 3756
W20250810 20:29:41.720301 3756 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.861s user 0.617s sys 1.241s
W20250810 20:29:41.723701 3765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:41.723977 3756 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.864s user 0.617s sys 1.241s
W20250810 20:29:41.727532 3764 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1863 milliseconds
I20250810 20:29:41.727622 3756 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:41.728919 3756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:41.731696 3756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:41.733152 3756 hybrid_clock.cc:648] HybridClock initialized: now 1754857781733109 us; error 56 us; skew 500 ppm
I20250810 20:29:41.734066 3756 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:41.741132 3756 webserver.cc:489] Webserver started at http://127.30.243.254:33545/ using document root <none> and password file <none>
I20250810 20:29:41.742172 3756 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:41.742410 3756 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:41.742895 3756 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:29:41.747655 3756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a552891af1fd429aaa1b73a2241cb902"
format_stamp: "Formatted at 2025-08-10 20:29:41 on dist-test-slave-7swh"
I20250810 20:29:41.748956 3756 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a552891af1fd429aaa1b73a2241cb902"
format_stamp: "Formatted at 2025-08-10 20:29:41 on dist-test-slave-7swh"
I20250810 20:29:41.757128 3756 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250810 20:29:41.763532 3772 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:41.764716 3756 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 20:29:41.765129 3756 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a552891af1fd429aaa1b73a2241cb902"
format_stamp: "Formatted at 2025-08-10 20:29:41 on dist-test-slave-7swh"
I20250810 20:29:41.765492 3756 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:41.820386 3756 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:41.822003 3756 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:41.822521 3756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:41.902180 3756 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:38889
I20250810 20:29:41.902278 3823 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:38889 every 8 connection(s)
I20250810 20:29:41.905063 3756 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 20:29:41.910509 3824 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:29:41.915793 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 3756
I20250810 20:29:41.916368 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 20:29:41.937151 3824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902: Bootstrap starting.
I20250810 20:29:41.943135 3824 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902: Neither blocks nor log segments found. Creating new log.
I20250810 20:29:41.945487 3824 log.cc:826] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902: Log is configured to *not* fsync() on all Append() calls
I20250810 20:29:41.950605 3824 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902: No bootstrap required, opened a new log
I20250810 20:29:41.969724 3824 raft_consensus.cc:357] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552891af1fd429aaa1b73a2241cb902" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 38889 } }
I20250810 20:29:41.970423 3824 raft_consensus.cc:383] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:29:41.970710 3824 raft_consensus.cc:738] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a552891af1fd429aaa1b73a2241cb902, State: Initialized, Role: FOLLOWER
I20250810 20:29:41.971550 3824 consensus_queue.cc:260] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552891af1fd429aaa1b73a2241cb902" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 38889 } }
I20250810 20:29:41.972074 3824 raft_consensus.cc:397] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:29:41.972354 3824 raft_consensus.cc:491] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:29:41.972710 3824 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:29:41.977051 3824 raft_consensus.cc:513] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552891af1fd429aaa1b73a2241cb902" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 38889 } }
I20250810 20:29:41.978000 3824 leader_election.cc:304] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a552891af1fd429aaa1b73a2241cb902; no voters:
I20250810 20:29:41.979833 3824 leader_election.cc:290] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:29:41.980616 3829 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:29:41.983002 3829 raft_consensus.cc:695] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [term 1 LEADER]: Becoming Leader. State: Replica: a552891af1fd429aaa1b73a2241cb902, State: Running, Role: LEADER
I20250810 20:29:41.983896 3829 consensus_queue.cc:237] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552891af1fd429aaa1b73a2241cb902" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 38889 } }
I20250810 20:29:41.985064 3824 sys_catalog.cc:564] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:29:41.994343 3830 sys_catalog.cc:455] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a552891af1fd429aaa1b73a2241cb902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552891af1fd429aaa1b73a2241cb902" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 38889 } } }
I20250810 20:29:41.994993 3830 sys_catalog.cc:458] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [sys.catalog]: This master's current role is: LEADER
I20250810 20:29:41.995918 3831 sys_catalog.cc:455] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a552891af1fd429aaa1b73a2241cb902. Latest consensus state: current_term: 1 leader_uuid: "a552891af1fd429aaa1b73a2241cb902" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a552891af1fd429aaa1b73a2241cb902" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 38889 } } }
I20250810 20:29:41.997896 3831 sys_catalog.cc:458] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902 [sys.catalog]: This master's current role is: LEADER
I20250810 20:29:42.001408 3837 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:29:42.016924 3837 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:29:42.034469 3837 catalog_manager.cc:1349] Generated new cluster ID: 06a21426c64b4fb984a07fbb609c047d
I20250810 20:29:42.034816 3837 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:29:42.054198 3837 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 20:29:42.056190 3837 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:29:42.068434 3837 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a552891af1fd429aaa1b73a2241cb902: Generated new TSK 0
I20250810 20:29:42.069715 3837 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 20:29:42.095000 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:0
--local_ip_for_outbound_sockets=127.30.243.193
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:38889
--builtin_ntp_servers=127.30.243.212:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250810 20:29:42.446056 3848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:42.446588 3848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:42.447175 3848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:42.481449 3848 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 20:29:42.481906 3848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:42.482714 3848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:29:42.522845 3848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:38889
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:42.524304 3848 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:42.526073 3848 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:42.541877 3854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:42.544848 3855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:42.546506 3857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:43.767374 3856 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250810 20:29:43.767910 3848 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:43.772306 3848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:43.775256 3848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:43.776815 3848 hybrid_clock.cc:648] HybridClock initialized: now 1754857783776775 us; error 37 us; skew 500 ppm
I20250810 20:29:43.777943 3848 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:43.785796 3848 webserver.cc:489] Webserver started at http://127.30.243.193:39733/ using document root <none> and password file <none>
I20250810 20:29:43.787163 3848 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:43.787447 3848 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:43.788038 3848 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:29:43.794854 3848 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "aa67135f4a6340f8a92f754760391412"
format_stamp: "Formatted at 2025-08-10 20:29:43 on dist-test-slave-7swh"
I20250810 20:29:43.796482 3848 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "aa67135f4a6340f8a92f754760391412"
format_stamp: "Formatted at 2025-08-10 20:29:43 on dist-test-slave-7swh"
I20250810 20:29:43.807845 3848 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.011s sys 0.000s
I20250810 20:29:43.816159 3864 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:43.817507 3848 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250810 20:29:43.817993 3848 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "aa67135f4a6340f8a92f754760391412"
format_stamp: "Formatted at 2025-08-10 20:29:43 on dist-test-slave-7swh"
I20250810 20:29:43.818444 3848 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:43.888000 3848 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:43.889516 3848 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:43.889940 3848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:43.892537 3848 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:29:43.896646 3848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:29:43.896862 3848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:43.897069 3848 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:29:43.897202 3848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:44.039593 3848 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:35013
I20250810 20:29:44.039710 3976 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:35013 every 8 connection(s)
I20250810 20:29:44.042255 3848 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 20:29:44.046645 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 3848
I20250810 20:29:44.047192 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 20:29:44.054143 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.194:0
--local_ip_for_outbound_sockets=127.30.243.194
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:38889
--builtin_ntp_servers=127.30.243.212:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250810 20:29:44.064275 3977 heartbeater.cc:344] Connected to a master server at 127.30.243.254:38889
I20250810 20:29:44.064733 3977 heartbeater.cc:461] Registering TS with master...
I20250810 20:29:44.065826 3977 heartbeater.cc:507] Master 127.30.243.254:38889 requested a full tablet report, sending...
I20250810 20:29:44.068652 3789 ts_manager.cc:194] Registered new tserver with Master: aa67135f4a6340f8a92f754760391412 (127.30.243.193:35013)
I20250810 20:29:44.070669 3789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:44557
W20250810 20:29:44.368326 3981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:44.368906 3981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:44.369609 3981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:44.402383 3981 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 20:29:44.402801 3981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:44.403640 3981 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.194
I20250810 20:29:44.439270 3981 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:38889
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.194
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:44.440753 3981 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:44.442454 3981 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:44.454571 3987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:29:45.074386 3977 heartbeater.cc:499] Master 127.30.243.254:38889 was elected leader, sending a full tablet report...
W20250810 20:29:44.456014 3988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:44.460189 3990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:45.660452 3989 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1200 milliseconds
I20250810 20:29:45.660596 3981 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:45.661890 3981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:45.664873 3981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:45.666364 3981 hybrid_clock.cc:648] HybridClock initialized: now 1754857785666311 us; error 70 us; skew 500 ppm
I20250810 20:29:45.667254 3981 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:45.674049 3981 webserver.cc:489] Webserver started at http://127.30.243.194:37309/ using document root <none> and password file <none>
I20250810 20:29:45.675238 3981 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:45.675473 3981 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:45.675985 3981 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:29:45.680652 3981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "06b9d2a12057407590d222e56b502ff9"
format_stamp: "Formatted at 2025-08-10 20:29:45 on dist-test-slave-7swh"
I20250810 20:29:45.681864 3981 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "06b9d2a12057407590d222e56b502ff9"
format_stamp: "Formatted at 2025-08-10 20:29:45 on dist-test-slave-7swh"
I20250810 20:29:45.689746 3981 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250810 20:29:45.695873 3997 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:45.697039 3981 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250810 20:29:45.697386 3981 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "06b9d2a12057407590d222e56b502ff9"
format_stamp: "Formatted at 2025-08-10 20:29:45 on dist-test-slave-7swh"
I20250810 20:29:45.697759 3981 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:45.754312 3981 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:45.755952 3981 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:45.756464 3981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:45.759194 3981 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:29:45.763526 3981 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:29:45.763779 3981 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:45.764096 3981 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:29:45.764257 3981 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:45.909324 3981 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.194:33421
I20250810 20:29:45.909444 4111 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.194:33421 every 8 connection(s)
I20250810 20:29:45.912133 3981 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250810 20:29:45.921185 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 3981
I20250810 20:29:45.921695 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250810 20:29:45.928743 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.195:0
--local_ip_for_outbound_sockets=127.30.243.195
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:38889
--builtin_ntp_servers=127.30.243.212:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250810 20:29:45.936617 4112 heartbeater.cc:344] Connected to a master server at 127.30.243.254:38889
I20250810 20:29:45.937209 4112 heartbeater.cc:461] Registering TS with master...
I20250810 20:29:45.938660 4112 heartbeater.cc:507] Master 127.30.243.254:38889 requested a full tablet report, sending...
I20250810 20:29:45.941465 3789 ts_manager.cc:194] Registered new tserver with Master: 06b9d2a12057407590d222e56b502ff9 (127.30.243.194:33421)
I20250810 20:29:45.942725 3789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.194:51141
W20250810 20:29:46.247179 4116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:46.247752 4116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:46.248306 4116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:46.281014 4116 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 20:29:46.281483 4116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:46.282300 4116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.195
I20250810 20:29:46.318756 4116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:38889
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.195
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:46.320196 4116 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:46.321868 4116 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:46.334803 4122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:29:46.946147 4112 heartbeater.cc:499] Master 127.30.243.254:38889 was elected leader, sending a full tablet report...
W20250810 20:29:46.336330 4123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:47.604849 4126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:47.607182 4125 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1268 milliseconds
I20250810 20:29:47.607373 4116 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:47.608633 4116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:47.610875 4116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:47.612322 4116 hybrid_clock.cc:648] HybridClock initialized: now 1754857787612278 us; error 64 us; skew 500 ppm
I20250810 20:29:47.613173 4116 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:47.619537 4116 webserver.cc:489] Webserver started at http://127.30.243.195:41625/ using document root <none> and password file <none>
I20250810 20:29:47.620530 4116 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:47.620735 4116 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:47.621229 4116 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:29:47.625782 4116 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "996082023ed847668277b222e2c8ffbe"
format_stamp: "Formatted at 2025-08-10 20:29:47 on dist-test-slave-7swh"
I20250810 20:29:47.627055 4116 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "996082023ed847668277b222e2c8ffbe"
format_stamp: "Formatted at 2025-08-10 20:29:47 on dist-test-slave-7swh"
I20250810 20:29:47.634651 4116 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250810 20:29:47.640408 4133 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:47.641450 4116 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250810 20:29:47.641774 4116 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "996082023ed847668277b222e2c8ffbe"
format_stamp: "Formatted at 2025-08-10 20:29:47 on dist-test-slave-7swh"
I20250810 20:29:47.642107 4116 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:47.700563 4116 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:47.702239 4116 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:47.702699 4116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:47.705328 4116 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:29:47.709770 4116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:29:47.709993 4116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:47.710301 4116 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:29:47.710462 4116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:47.854688 4116 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.195:45955
I20250810 20:29:47.854831 4246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.195:45955 every 8 connection(s)
I20250810 20:29:47.857383 4116 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250810 20:29:47.861393 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 4116
I20250810 20:29:47.861933 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250810 20:29:47.869136 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.196:0
--local_ip_for_outbound_sockets=127.30.243.196
--webserver_interface=127.30.243.196
--webserver_port=0
--tserver_master_addrs=127.30.243.254:38889
--builtin_ntp_servers=127.30.243.212:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250810 20:29:47.884415 4247 heartbeater.cc:344] Connected to a master server at 127.30.243.254:38889
I20250810 20:29:47.884903 4247 heartbeater.cc:461] Registering TS with master...
I20250810 20:29:47.885983 4247 heartbeater.cc:507] Master 127.30.243.254:38889 requested a full tablet report, sending...
I20250810 20:29:47.888454 3789 ts_manager.cc:194] Registered new tserver with Master: 996082023ed847668277b222e2c8ffbe (127.30.243.195:45955)
I20250810 20:29:47.890416 3789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.195:49653
W20250810 20:29:48.201198 4251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:48.201778 4251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:48.202306 4251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:48.234699 4251 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 20:29:48.235391 4251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:48.236318 4251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.196
I20250810 20:29:48.272408 4251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.30.243.196
--webserver_port=0
--tserver_master_addrs=127.30.243.254:38889
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.196
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:48.273690 4251 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:48.275391 4251 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:48.289008 4257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:29:48.894284 4247 heartbeater.cc:499] Master 127.30.243.254:38889 was elected leader, sending a full tablet report...
W20250810 20:29:49.692081 4256 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 4251
W20250810 20:29:49.770027 4251 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.481s user 0.510s sys 0.933s
W20250810 20:29:49.770466 4251 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.482s user 0.510s sys 0.933s
W20250810 20:29:48.289511 4258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:49.773330 4260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:49.776245 4259 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1483 milliseconds
I20250810 20:29:49.776320 4251 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:49.777622 4251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:49.781332 4251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:49.782842 4251 hybrid_clock.cc:648] HybridClock initialized: now 1754857789782799 us; error 56 us; skew 500 ppm
I20250810 20:29:49.783772 4251 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:49.790045 4251 webserver.cc:489] Webserver started at http://127.30.243.196:43223/ using document root <none> and password file <none>
I20250810 20:29:49.791095 4251 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:49.791323 4251 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:49.791787 4251 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:29:49.796308 4251 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "dad18b517c804d7d9b8fcc8caa8bbde5"
format_stamp: "Formatted at 2025-08-10 20:29:49 on dist-test-slave-7swh"
I20250810 20:29:49.797508 4251 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "dad18b517c804d7d9b8fcc8caa8bbde5"
format_stamp: "Formatted at 2025-08-10 20:29:49 on dist-test-slave-7swh"
I20250810 20:29:49.805572 4251 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20250810 20:29:49.811657 4269 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:49.812774 4251 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250810 20:29:49.813167 4251 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "dad18b517c804d7d9b8fcc8caa8bbde5"
format_stamp: "Formatted at 2025-08-10 20:29:49 on dist-test-slave-7swh"
I20250810 20:29:49.813525 4251 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:49.864341 4251 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:49.865965 4251 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:49.866428 4251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:49.869180 4251 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:29:49.873451 4251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:29:49.873704 4251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:49.873975 4251 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:29:49.874138 4251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:50.026451 4251 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.196:41581
I20250810 20:29:50.026566 4382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.196:41581 every 8 connection(s)
I20250810 20:29:50.029215 4251 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250810 20:29:50.033916 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 4251
I20250810 20:29:50.034658 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250810 20:29:50.042528 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.197:0
--local_ip_for_outbound_sockets=127.30.243.197
--webserver_interface=127.30.243.197
--webserver_port=0
--tserver_master_addrs=127.30.243.254:38889
--builtin_ntp_servers=127.30.243.212:38377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250810 20:29:50.056152 4383 heartbeater.cc:344] Connected to a master server at 127.30.243.254:38889
I20250810 20:29:50.056627 4383 heartbeater.cc:461] Registering TS with master...
I20250810 20:29:50.057749 4383 heartbeater.cc:507] Master 127.30.243.254:38889 requested a full tablet report, sending...
I20250810 20:29:50.060366 3789 ts_manager.cc:194] Registered new tserver with Master: dad18b517c804d7d9b8fcc8caa8bbde5 (127.30.243.196:41581)
I20250810 20:29:50.061807 3789 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.196:42157
W20250810 20:29:50.367494 4387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:29:50.368024 4387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:29:50.368490 4387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:29:50.401737 4387 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250810 20:29:50.402117 4387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:29:50.402958 4387 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.197
I20250810 20:29:50.438464 4387 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:38377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.30.243.197
--webserver_port=0
--tserver_master_addrs=127.30.243.254:38889
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.197
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:29:50.439822 4387 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:29:50.441474 4387 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:29:50.454274 4393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:29:51.065407 4383 heartbeater.cc:499] Master 127.30.243.254:38889 was elected leader, sending a full tablet report...
W20250810 20:29:50.456152 4394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:50.460765 4396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:29:51.701434 4395 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1241 milliseconds
I20250810 20:29:51.701581 4387 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:29:51.702862 4387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:29:51.705101 4387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:29:51.706480 4387 hybrid_clock.cc:648] HybridClock initialized: now 1754857791706450 us; error 35 us; skew 500 ppm
I20250810 20:29:51.707367 4387 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:29:51.714216 4387 webserver.cc:489] Webserver started at http://127.30.243.197:33643/ using document root <none> and password file <none>
I20250810 20:29:51.715205 4387 fs_manager.cc:362] Metadata directory not provided
I20250810 20:29:51.715405 4387 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:29:51.715830 4387 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:29:51.720310 4387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "58d04901615e41048b18ba86c58151a1"
format_stamp: "Formatted at 2025-08-10 20:29:51 on dist-test-slave-7swh"
I20250810 20:29:51.721424 4387 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "58d04901615e41048b18ba86c58151a1"
format_stamp: "Formatted at 2025-08-10 20:29:51 on dist-test-slave-7swh"
I20250810 20:29:51.729826 4387 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.001s sys 0.008s
I20250810 20:29:51.736783 4403 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:51.738116 4387 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20250810 20:29:51.738445 4387 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "58d04901615e41048b18ba86c58151a1"
format_stamp: "Formatted at 2025-08-10 20:29:51 on dist-test-slave-7swh"
I20250810 20:29:51.738761 4387 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:29:51.797899 4387 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:29:51.799579 4387 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:29:51.800026 4387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:29:51.803372 4387 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:29:51.807929 4387 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:29:51.808161 4387 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:51.808378 4387 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:29:51.808522 4387 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:29:52.005853 4387 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.197:32771
I20250810 20:29:52.005940 4516 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.197:32771 every 8 connection(s)
I20250810 20:29:52.008491 4387 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250810 20:29:52.011663 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 4387
I20250810 20:29:52.012250 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250810 20:29:52.033315 4517 heartbeater.cc:344] Connected to a master server at 127.30.243.254:38889
I20250810 20:29:52.033867 4517 heartbeater.cc:461] Registering TS with master...
I20250810 20:29:52.035645 4517 heartbeater.cc:507] Master 127.30.243.254:38889 requested a full tablet report, sending...
I20250810 20:29:52.038915 3788 ts_manager.cc:194] Registered new tserver with Master: 58d04901615e41048b18ba86c58151a1 (127.30.243.197:32771)
I20250810 20:29:52.041056 3788 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.197:51315
I20250810 20:29:52.050228 31695 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250810 20:29:52.091965 3788 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49164:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250810 20:29:52.183799 4046 tablet_service.cc:1468] Processing CreateTablet for tablet 4c22b2b2026d49a6a24d2237acbe03cd (DEFAULT_TABLE table=TestTable [id=2b87ae4619d84215b8c377909bb928d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:29:52.188477 4046 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c22b2b2026d49a6a24d2237acbe03cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:29:52.187911 3912 tablet_service.cc:1468] Processing CreateTablet for tablet 4c22b2b2026d49a6a24d2237acbe03cd (DEFAULT_TABLE table=TestTable [id=2b87ae4619d84215b8c377909bb928d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:29:52.189894 3912 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c22b2b2026d49a6a24d2237acbe03cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:29:52.191826 4451 tablet_service.cc:1468] Processing CreateTablet for tablet 4c22b2b2026d49a6a24d2237acbe03cd (DEFAULT_TABLE table=TestTable [id=2b87ae4619d84215b8c377909bb928d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:29:52.193890 4451 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c22b2b2026d49a6a24d2237acbe03cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:29:52.217820 4536 tablet_bootstrap.cc:492] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9: Bootstrap starting.
I20250810 20:29:52.221987 4537 tablet_bootstrap.cc:492] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412: Bootstrap starting.
I20250810 20:29:52.228577 4538 tablet_bootstrap.cc:492] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1: Bootstrap starting.
I20250810 20:29:52.228335 4536 tablet_bootstrap.cc:654] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9: Neither blocks nor log segments found. Creating new log.
I20250810 20:29:52.231204 4536 log.cc:826] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9: Log is configured to *not* fsync() on all Append() calls
I20250810 20:29:52.231765 4537 tablet_bootstrap.cc:654] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412: Neither blocks nor log segments found. Creating new log.
I20250810 20:29:52.236236 4537 log.cc:826] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412: Log is configured to *not* fsync() on all Append() calls
I20250810 20:29:52.238245 4536 tablet_bootstrap.cc:492] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9: No bootstrap required, opened a new log
I20250810 20:29:52.238750 4536 ts_tablet_manager.cc:1397] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.006s
I20250810 20:29:52.239313 4538 tablet_bootstrap.cc:654] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1: Neither blocks nor log segments found. Creating new log.
I20250810 20:29:52.242486 4538 log.cc:826] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1: Log is configured to *not* fsync() on all Append() calls
I20250810 20:29:52.245182 4537 tablet_bootstrap.cc:492] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412: No bootstrap required, opened a new log
I20250810 20:29:52.245797 4537 ts_tablet_manager.cc:1397] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412: Time spent bootstrapping tablet: real 0.025s user 0.007s sys 0.012s
I20250810 20:29:52.248538 4538 tablet_bootstrap.cc:492] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1: No bootstrap required, opened a new log
I20250810 20:29:52.249178 4538 ts_tablet_manager.cc:1397] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.002s
I20250810 20:29:52.268592 4537 raft_consensus.cc:357] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 } } peers { permanent_uuid: "58d04901615e41048b18ba86c58151a1" member_type: VOTER last_known_addr { host: "127.30.243.197" port: 32771 } } peers { permanent_uuid: "aa67135f4a6340f8a92f754760391412" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 35013 } }
I20250810 20:29:52.269562 4537 raft_consensus.cc:383] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:29:52.269836 4537 raft_consensus.cc:738] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa67135f4a6340f8a92f754760391412, State: Initialized, Role: FOLLOWER
I20250810 20:29:52.269944 4536 raft_consensus.cc:357] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 } } peers { permanent_uuid: "58d04901615e41048b18ba86c58151a1" member_type: VOTER last_known_addr { host: "127.30.243.197" port: 32771 } } peers { permanent_uuid: "aa67135f4a6340f8a92f754760391412" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 35013 } }
I20250810 20:29:52.270970 4536 raft_consensus.cc:383] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:29:52.271332 4536 raft_consensus.cc:738] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06b9d2a12057407590d222e56b502ff9, State: Initialized, Role: FOLLOWER
I20250810 20:29:52.271198 4537 consensus_queue.cc:260] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 } } peers { permanent_uuid: "58d04901615e41048b18ba86c58151a1" member_type: VOTER last_known_addr { host: "127.30.243.197" port: 32771 } } peers { permanent_uuid: "aa67135f4a6340f8a92f754760391412" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 35013 } }
I20250810 20:29:52.272301 4536 consensus_queue.cc:260] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 } } peers { permanent_uuid: "58d04901615e41048b18ba86c58151a1" member_type: VOTER last_known_addr { host: "127.30.243.197" port: 32771 } } peers { permanent_uuid: "aa67135f4a6340f8a92f754760391412" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 35013 } }
I20250810 20:29:52.276376 4537 ts_tablet_manager.cc:1428] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412: Time spent starting tablet: real 0.030s user 0.028s sys 0.004s
I20250810 20:29:52.280889 4538 raft_consensus.cc:357] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 } } peers { permanent_uuid: "58d04901615e41048b18ba86c58151a1" member_type: VOTER last_known_addr { host: "127.30.243.197" port: 32771 } } peers { permanent_uuid: "aa67135f4a6340f8a92f754760391412" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 35013 } }
I20250810 20:29:52.282009 4538 raft_consensus.cc:383] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:29:52.279436 4536 ts_tablet_manager.cc:1428] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9: Time spent starting tablet: real 0.040s user 0.028s sys 0.012s
I20250810 20:29:52.282351 4538 raft_consensus.cc:738] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58d04901615e41048b18ba86c58151a1, State: Initialized, Role: FOLLOWER
I20250810 20:29:52.283360 4538 consensus_queue.cc:260] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 } } peers { permanent_uuid: "58d04901615e41048b18ba86c58151a1" member_type: VOTER last_known_addr { host: "127.30.243.197" port: 32771 } } peers { permanent_uuid: "aa67135f4a6340f8a92f754760391412" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 35013 } }
I20250810 20:29:52.287808 4517 heartbeater.cc:499] Master 127.30.243.254:38889 was elected leader, sending a full tablet report...
I20250810 20:29:52.288938 4538 ts_tablet_manager.cc:1428] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1: Time spent starting tablet: real 0.039s user 0.032s sys 0.007s
W20250810 20:29:52.308455 3978 tablet.cc:2378] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:29:52.399645 4544 raft_consensus.cc:491] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:29:52.400257 4544 raft_consensus.cc:513] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 } } peers { permanent_uuid: "58d04901615e41048b18ba86c58151a1" member_type: VOTER last_known_addr { host: "127.30.243.197" port: 32771 } } peers { permanent_uuid: "aa67135f4a6340f8a92f754760391412" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 35013 } }
I20250810 20:29:52.402901 4544 leader_election.cc:290] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06b9d2a12057407590d222e56b502ff9 (127.30.243.194:33421), aa67135f4a6340f8a92f754760391412 (127.30.243.193:35013)
I20250810 20:29:52.413266 4066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c22b2b2026d49a6a24d2237acbe03cd" candidate_uuid: "58d04901615e41048b18ba86c58151a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b9d2a12057407590d222e56b502ff9" is_pre_election: true
I20250810 20:29:52.414598 4066 raft_consensus.cc:2466] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d04901615e41048b18ba86c58151a1 in term 0.
I20250810 20:29:52.415309 3932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c22b2b2026d49a6a24d2237acbe03cd" candidate_uuid: "58d04901615e41048b18ba86c58151a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67135f4a6340f8a92f754760391412" is_pre_election: true
I20250810 20:29:52.416146 4405 leader_election.cc:304] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b9d2a12057407590d222e56b502ff9, 58d04901615e41048b18ba86c58151a1; no voters:
I20250810 20:29:52.416316 3932 raft_consensus.cc:2466] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58d04901615e41048b18ba86c58151a1 in term 0.
I20250810 20:29:52.417088 4544 raft_consensus.cc:2802] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 20:29:52.417469 4544 raft_consensus.cc:491] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 20:29:52.417846 4544 raft_consensus.cc:3058] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 0 FOLLOWER]: Advancing to term 1
W20250810 20:29:52.424983 4113 tablet.cc:2378] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:29:52.427018 4544 raft_consensus.cc:513] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 } } peers { permanent_uuid: "58d04901615e41048b18ba86c58151a1" member_type: VOTER last_known_addr { host: "127.30.243.197" port: 32771 } } peers { permanent_uuid: "aa67135f4a6340f8a92f754760391412" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 35013 } }
I20250810 20:29:52.431025 4544 leader_election.cc:290] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [CANDIDATE]: Term 1 election: Requested vote from peers 06b9d2a12057407590d222e56b502ff9 (127.30.243.194:33421), aa67135f4a6340f8a92f754760391412 (127.30.243.193:35013)
I20250810 20:29:52.432055 4066 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c22b2b2026d49a6a24d2237acbe03cd" candidate_uuid: "58d04901615e41048b18ba86c58151a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06b9d2a12057407590d222e56b502ff9"
I20250810 20:29:52.432504 4066 raft_consensus.cc:3058] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:29:52.432498 3932 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c22b2b2026d49a6a24d2237acbe03cd" candidate_uuid: "58d04901615e41048b18ba86c58151a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa67135f4a6340f8a92f754760391412"
I20250810 20:29:52.433324 3932 raft_consensus.cc:3058] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:29:52.437525 4066 raft_consensus.cc:2466] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d04901615e41048b18ba86c58151a1 in term 1.
I20250810 20:29:52.438416 3932 raft_consensus.cc:2466] T 4c22b2b2026d49a6a24d2237acbe03cd P aa67135f4a6340f8a92f754760391412 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58d04901615e41048b18ba86c58151a1 in term 1.
I20250810 20:29:52.438547 4405 leader_election.cc:304] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06b9d2a12057407590d222e56b502ff9, 58d04901615e41048b18ba86c58151a1; no voters:
I20250810 20:29:52.439404 4544 raft_consensus.cc:2802] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:29:52.441022 4544 raft_consensus.cc:695] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [term 1 LEADER]: Becoming Leader. State: Replica: 58d04901615e41048b18ba86c58151a1, State: Running, Role: LEADER
I20250810 20:29:52.441921 4544 consensus_queue.cc:237] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 } } peers { permanent_uuid: "58d04901615e41048b18ba86c58151a1" member_type: VOTER last_known_addr { host: "127.30.243.197" port: 32771 } } peers { permanent_uuid: "aa67135f4a6340f8a92f754760391412" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 35013 } }
I20250810 20:29:52.453184 3787 catalog_manager.cc:5582] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 58d04901615e41048b18ba86c58151a1 (127.30.243.197). New cstate: current_term: 1 leader_uuid: "58d04901615e41048b18ba86c58151a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58d04901615e41048b18ba86c58151a1" member_type: VOTER last_known_addr { host: "127.30.243.197" port: 32771 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aa67135f4a6340f8a92f754760391412" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 35013 } health_report { overall_health: UNKNOWN } } }
I20250810 20:29:52.491986 31695 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250810 20:29:52.496587 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver aa67135f4a6340f8a92f754760391412 to finish bootstrapping
W20250810 20:29:52.514470 4518 tablet.cc:2378] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:29:52.514899 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 06b9d2a12057407590d222e56b502ff9 to finish bootstrapping
I20250810 20:29:52.528981 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 58d04901615e41048b18ba86c58151a1 to finish bootstrapping
I20250810 20:29:52.541769 31695 test_util.cc:276] Using random seed: -1420094422
I20250810 20:29:52.568351 31695 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250810 20:29:52.569322 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 3848
I20250810 20:29:52.605790 4066 raft_consensus.cc:1273] T 4c22b2b2026d49a6a24d2237acbe03cd P 06b9d2a12057407590d222e56b502ff9 [term 1 FOLLOWER]: Refusing update from remote peer 58d04901615e41048b18ba86c58151a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250810 20:29:52.607299 4406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.243.193:35013: connect: Connection refused (error 111)
I20250810 20:29:52.607756 4547 consensus_queue.cc:1035] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06b9d2a12057407590d222e56b502ff9" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 33421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250810 20:29:52.612929 4406 consensus_peers.cc:489] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 -> Peer aa67135f4a6340f8a92f754760391412 (127.30.243.193:35013): Couldn't send request to peer aa67135f4a6340f8a92f754760391412. Status: Network error: Client connection negotiation failed: client connection to 127.30.243.193:35013: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250810 20:29:52.628664 4557 mvcc.cc:204] Tried to move back new op lower bound from 7187897518492766208 to 7187897517850292224. Current Snapshot: MvccSnapshot[applied={T|T < 7187897518492766208}]
I20250810 20:29:52.633666 4560 mvcc.cc:204] Tried to move back new op lower bound from 7187897518492766208 to 7187897517850292224. Current Snapshot: MvccSnapshot[applied={T|T < 7187897518492766208}]
W20250810 20:29:55.065136 4406 consensus_peers.cc:489] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 -> Peer aa67135f4a6340f8a92f754760391412 (127.30.243.193:35013): Couldn't send request to peer aa67135f4a6340f8a92f754760391412. Status: Network error: Client connection negotiation failed: client connection to 127.30.243.193:35013: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250810 20:29:55.462613 4046 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 20:29:55.478363 4182 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 20:29:55.502898 4451 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 20:29:55.539341 4318 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 20:29:57.376448 4046 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 20:29:57.393095 4451 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 20:29:57.402271 4182 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 20:29:57.428148 4318 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250810 20:29:57.928472 4406 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.243.193:35013: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250810 20:29:57.940994 4406 consensus_peers.cc:489] T 4c22b2b2026d49a6a24d2237acbe03cd P 58d04901615e41048b18ba86c58151a1 -> Peer aa67135f4a6340f8a92f754760391412 (127.30.243.193:35013): Couldn't send request to peer aa67135f4a6340f8a92f754760391412. Status: Network error: Client connection negotiation failed: client connection to 127.30.243.193:35013: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250810 20:29:59.824270 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 3981
I20250810 20:29:59.863046 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 4116
I20250810 20:29:59.887527 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 4251
I20250810 20:29:59.914624 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 4387
I20250810 20:29:59.958882 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 3756
2025-08-10T20:29:59Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (20629 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (20629 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250810 20:30:00.026479 31695 test_util.cc:276] Using random seed: -1412609715
I20250810 20:30:00.030862 31695 ts_itest-base.cc:115] Starting cluster with:
I20250810 20:30:00.031111 31695 ts_itest-base.cc:116] --------------
I20250810 20:30:00.031241 31695 ts_itest-base.cc:117] 1 tablet servers
I20250810 20:30:00.031363 31695 ts_itest-base.cc:118] 1 replicas per TS
I20250810 20:30:00.031484 31695 ts_itest-base.cc:119] --------------
2025-08-10T20:30:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T20:30:00Z Disabled control of system clock
I20250810 20:30:00.080611 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:36221
--webserver_interface=127.30.243.254
--webserver_port=0
--builtin_ntp_servers=127.30.243.212:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:36221 with env {}
W20250810 20:30:00.409121 4666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:00.409682 4666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:30:00.410105 4666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:00.442533 4666 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:30:00.442853 4666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:30:00.443114 4666 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:30:00.443323 4666 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:30:00.480199 4666 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:36221
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:36221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:30:00.481492 4666 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:00.483342 4666 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:00.494872 4672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:00.496201 4673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:00.500901 4675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:01.717118 4674 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250810 20:30:01.717286 4666 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:30:01.721606 4666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:30:01.725262 4666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:30:01.726804 4666 hybrid_clock.cc:648] HybridClock initialized: now 1754857801726712 us; error 96 us; skew 500 ppm
I20250810 20:30:01.727838 4666 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:01.734642 4666 webserver.cc:489] Webserver started at http://127.30.243.254:33095/ using document root <none> and password file <none>
I20250810 20:30:01.735709 4666 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:01.735950 4666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:01.736408 4666 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:30:01.741402 4666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "67ae272d0099440fa263bf7c737a673c"
format_stamp: "Formatted at 2025-08-10 20:30:01 on dist-test-slave-7swh"
I20250810 20:30:01.742924 4666 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "67ae272d0099440fa263bf7c737a673c"
format_stamp: "Formatted at 2025-08-10 20:30:01 on dist-test-slave-7swh"
I20250810 20:30:01.752189 4666 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.003s sys 0.005s
I20250810 20:30:01.758456 4682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:01.760301 4666 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250810 20:30:01.760665 4666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
uuid: "67ae272d0099440fa263bf7c737a673c"
format_stamp: "Formatted at 2025-08-10 20:30:01 on dist-test-slave-7swh"
I20250810 20:30:01.761020 4666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:01.815788 4666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:01.817353 4666 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:01.817795 4666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:01.914320 4666 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:36221
I20250810 20:30:01.914395 4733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:36221 every 8 connection(s)
I20250810 20:30:01.917191 4666 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250810 20:30:01.922793 4734 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:30:01.926151 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 4666
I20250810 20:30:01.926785 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250810 20:30:01.947005 4734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c: Bootstrap starting.
I20250810 20:30:01.953636 4734 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c: Neither blocks nor log segments found. Creating new log.
I20250810 20:30:01.955796 4734 log.cc:826] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c: Log is configured to *not* fsync() on all Append() calls
I20250810 20:30:01.962618 4734 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c: No bootstrap required, opened a new log
I20250810 20:30:01.981266 4734 raft_consensus.cc:357] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ae272d0099440fa263bf7c737a673c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36221 } }
I20250810 20:30:01.982064 4734 raft_consensus.cc:383] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:30:01.982393 4734 raft_consensus.cc:738] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67ae272d0099440fa263bf7c737a673c, State: Initialized, Role: FOLLOWER
I20250810 20:30:01.983362 4734 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ae272d0099440fa263bf7c737a673c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36221 } }
I20250810 20:30:01.983932 4734 raft_consensus.cc:397] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:30:01.984243 4734 raft_consensus.cc:491] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:30:01.984591 4734 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:30:01.988999 4734 raft_consensus.cc:513] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ae272d0099440fa263bf7c737a673c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36221 } }
I20250810 20:30:01.989765 4734 leader_election.cc:304] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67ae272d0099440fa263bf7c737a673c; no voters:
I20250810 20:30:01.991715 4734 leader_election.cc:290] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:30:01.992506 4739 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:30:01.994758 4739 raft_consensus.cc:695] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [term 1 LEADER]: Becoming Leader. State: Replica: 67ae272d0099440fa263bf7c737a673c, State: Running, Role: LEADER
I20250810 20:30:01.995754 4739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ae272d0099440fa263bf7c737a673c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36221 } }
I20250810 20:30:01.996759 4734 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:30:02.006989 4740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67ae272d0099440fa263bf7c737a673c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ae272d0099440fa263bf7c737a673c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36221 } } }
I20250810 20:30:02.007464 4741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67ae272d0099440fa263bf7c737a673c. Latest consensus state: current_term: 1 leader_uuid: "67ae272d0099440fa263bf7c737a673c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67ae272d0099440fa263bf7c737a673c" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36221 } } }
I20250810 20:30:02.008000 4741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [sys.catalog]: This master's current role is: LEADER
I20250810 20:30:02.007917 4740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c [sys.catalog]: This master's current role is: LEADER
I20250810 20:30:02.010802 4746 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:30:02.025552 4746 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:30:02.049345 4746 catalog_manager.cc:1349] Generated new cluster ID: 387145e3768948a3af5689bd8badacab
I20250810 20:30:02.049641 4746 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:30:02.072312 4746 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 20:30:02.074579 4746 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:30:02.093142 4746 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 67ae272d0099440fa263bf7c737a673c: Generated new TSK 0
I20250810 20:30:02.094372 4746 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 20:30:02.112706 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:0
--local_ip_for_outbound_sockets=127.30.243.193
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:36221
--builtin_ntp_servers=127.30.243.212:45149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250810 20:30:02.457597 4758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:02.458099 4758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:30:02.458549 4758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:02.491215 4758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:30:02.492038 4758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:30:02.534497 4758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:45149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:36221
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:30:02.535954 4758 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:02.537743 4758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:02.552567 4764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:02.554203 4765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:03.955513 4763 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 4758
W20250810 20:30:04.507992 4763 kernel_stack_watchdog.cc:198] Thread 4758 stuck at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 20:30:04.509964 4758 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.956s user 0.698s sys 1.193s
W20250810 20:30:04.510530 4758 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.957s user 0.698s sys 1.194s
W20250810 20:30:04.510973 4767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:04.512393 4766 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1954 milliseconds
I20250810 20:30:04.512470 4758 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:30:04.513602 4758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:30:04.516223 4758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:30:04.517665 4758 hybrid_clock.cc:648] HybridClock initialized: now 1754857804517604 us; error 77 us; skew 500 ppm
I20250810 20:30:04.518513 4758 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:04.525197 4758 webserver.cc:489] Webserver started at http://127.30.243.193:35451/ using document root <none> and password file <none>
I20250810 20:30:04.526160 4758 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:04.526383 4758 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:04.526823 4758 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:30:04.531813 4758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "2dbc9fa247a84626b6f11331a2489399"
format_stamp: "Formatted at 2025-08-10 20:30:04 on dist-test-slave-7swh"
I20250810 20:30:04.532985 4758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "2dbc9fa247a84626b6f11331a2489399"
format_stamp: "Formatted at 2025-08-10 20:30:04 on dist-test-slave-7swh"
I20250810 20:30:04.540604 4758 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250810 20:30:04.546772 4774 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:04.547950 4758 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250810 20:30:04.548271 4758 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "2dbc9fa247a84626b6f11331a2489399"
format_stamp: "Formatted at 2025-08-10 20:30:04 on dist-test-slave-7swh"
I20250810 20:30:04.548584 4758 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:04.611737 4758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:04.613236 4758 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:04.613703 4758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:04.616935 4758 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:30:04.621384 4758 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:30:04.621588 4758 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:04.621891 4758 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:30:04.622043 4758 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:04.805919 4758 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:44167
I20250810 20:30:04.806020 4888 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:44167 every 8 connection(s)
I20250810 20:30:04.808729 4758 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250810 20:30:04.819646 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 4758
I20250810 20:30:04.820212 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1754857648852986-31695-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250810 20:30:04.832659 4889 heartbeater.cc:344] Connected to a master server at 127.30.243.254:36221
I20250810 20:30:04.833209 4889 heartbeater.cc:461] Registering TS with master...
I20250810 20:30:04.834529 4889 heartbeater.cc:507] Master 127.30.243.254:36221 requested a full tablet report, sending...
I20250810 20:30:04.837563 4699 ts_manager.cc:194] Registered new tserver with Master: 2dbc9fa247a84626b6f11331a2489399 (127.30.243.193:44167)
I20250810 20:30:04.840155 4699 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:48485
I20250810 20:30:04.841373 31695 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250810 20:30:04.880322 4699 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37518:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250810 20:30:04.979519 4823 tablet_service.cc:1468] Processing CreateTablet for tablet 2316637c42d445579dfacef3145fae90 (DEFAULT_TABLE table=TestTable [id=d8e1986481a045c2803a5736d2a3917c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:30:04.981775 4823 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2316637c42d445579dfacef3145fae90. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:30:05.010967 4904 tablet_bootstrap.cc:492] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399: Bootstrap starting.
I20250810 20:30:05.017083 4904 tablet_bootstrap.cc:654] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399: Neither blocks nor log segments found. Creating new log.
I20250810 20:30:05.019135 4904 log.cc:826] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399: Log is configured to *not* fsync() on all Append() calls
I20250810 20:30:05.024595 4904 tablet_bootstrap.cc:492] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399: No bootstrap required, opened a new log
I20250810 20:30:05.025115 4904 ts_tablet_manager.cc:1397] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.011s
I20250810 20:30:05.043994 4904 raft_consensus.cc:357] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc9fa247a84626b6f11331a2489399" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44167 } }
I20250810 20:30:05.044839 4904 raft_consensus.cc:383] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:30:05.045197 4904 raft_consensus.cc:738] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dbc9fa247a84626b6f11331a2489399, State: Initialized, Role: FOLLOWER
I20250810 20:30:05.045993 4904 consensus_queue.cc:260] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc9fa247a84626b6f11331a2489399" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44167 } }
I20250810 20:30:05.046552 4904 raft_consensus.cc:397] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:30:05.046836 4904 raft_consensus.cc:491] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:30:05.047242 4904 raft_consensus.cc:3058] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:30:05.051947 4904 raft_consensus.cc:513] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc9fa247a84626b6f11331a2489399" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44167 } }
I20250810 20:30:05.052747 4904 leader_election.cc:304] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dbc9fa247a84626b6f11331a2489399; no voters:
I20250810 20:30:05.054788 4904 leader_election.cc:290] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:30:05.055241 4906 raft_consensus.cc:2802] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:30:05.059091 4889 heartbeater.cc:499] Master 127.30.243.254:36221 was elected leader, sending a full tablet report...
I20250810 20:30:05.060670 4904 ts_tablet_manager.cc:1428] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399: Time spent starting tablet: real 0.035s user 0.035s sys 0.001s
I20250810 20:30:05.060902 4906 raft_consensus.cc:695] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [term 1 LEADER]: Becoming Leader. State: Replica: 2dbc9fa247a84626b6f11331a2489399, State: Running, Role: LEADER
I20250810 20:30:05.061712 4906 consensus_queue.cc:237] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc9fa247a84626b6f11331a2489399" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44167 } }
W20250810 20:30:05.064383 4890 tablet.cc:2378] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:30:05.076205 4699 catalog_manager.cc:5582] T 2316637c42d445579dfacef3145fae90 P 2dbc9fa247a84626b6f11331a2489399 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2dbc9fa247a84626b6f11331a2489399 (127.30.243.193). New cstate: current_term: 1 leader_uuid: "2dbc9fa247a84626b6f11331a2489399" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dbc9fa247a84626b6f11331a2489399" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 44167 } health_report { overall_health: HEALTHY } } }
I20250810 20:30:05.102841 31695 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250810 20:30:05.106245 31695 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 2dbc9fa247a84626b6f11331a2489399 to finish bootstrapping
I20250810 20:30:07.913168 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 4758
I20250810 20:30:07.943159 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 4666
2025-08-10T20:30:07Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7975 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7976 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250810 20:30:08.003201 31695 test_util.cc:276] Using random seed: -1404632990
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (14 ms)
[----------] 1 test from ListTableCliParamTest (14 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-10T20:30:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-10T20:30:08Z Disabled control of system clock
I20250810 20:30:08.063997 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:36065
--webserver_interface=127.30.243.254
--webserver_port=0
--builtin_ntp_servers=127.30.243.212:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:36065 with env {}
W20250810 20:30:08.388075 4933 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:08.388717 4933 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:30:08.389214 4933 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:08.422088 4933 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:30:08.422425 4933 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:30:08.422685 4933 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:30:08.422979 4933 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:30:08.460023 4933 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:36065
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:36065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:30:08.461417 4933 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:08.463222 4933 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:08.474550 4939 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:08.476061 4940 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:09.730130 4942 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:09.732748 4941 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1252 milliseconds
I20250810 20:30:09.732942 4933 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:30:09.734265 4933 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:30:09.737123 4933 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:30:09.738529 4933 hybrid_clock.cc:648] HybridClock initialized: now 1754857809738467 us; error 52 us; skew 500 ppm
I20250810 20:30:09.739697 4933 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:09.746568 4933 webserver.cc:489] Webserver started at http://127.30.243.254:40573/ using document root <none> and password file <none>
I20250810 20:30:09.747720 4933 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:09.747967 4933 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:09.748445 4933 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:30:09.753190 4933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/instance:
uuid: "0dc8ae5752c5467da2df9907c1958e26"
format_stamp: "Formatted at 2025-08-10 20:30:09 on dist-test-slave-7swh"
I20250810 20:30:09.754393 4933 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal/instance:
uuid: "0dc8ae5752c5467da2df9907c1958e26"
format_stamp: "Formatted at 2025-08-10 20:30:09 on dist-test-slave-7swh"
I20250810 20:30:09.762174 4933 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250810 20:30:09.768102 4950 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:09.769483 4933 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250810 20:30:09.769825 4933 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
uuid: "0dc8ae5752c5467da2df9907c1958e26"
format_stamp: "Formatted at 2025-08-10 20:30:09 on dist-test-slave-7swh"
I20250810 20:30:09.770164 4933 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:09.843910 4933 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:09.845387 4933 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:09.845824 4933 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:09.920388 4933 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:36065
I20250810 20:30:09.920424 5002 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:36065 every 8 connection(s)
I20250810 20:30:09.923312 4933 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/info.pb
I20250810 20:30:09.928768 5003 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:30:09.932467 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 4933
I20250810 20:30:09.933120 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal/instance
I20250810 20:30:09.955485 5003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26: Bootstrap starting.
I20250810 20:30:09.960861 5003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26: Neither blocks nor log segments found. Creating new log.
I20250810 20:30:09.962672 5003 log.cc:826] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26: Log is configured to *not* fsync() on all Append() calls
I20250810 20:30:09.968477 5003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26: No bootstrap required, opened a new log
I20250810 20:30:09.987782 5003 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc8ae5752c5467da2df9907c1958e26" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } }
I20250810 20:30:09.988499 5003 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:30:09.988703 5003 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dc8ae5752c5467da2df9907c1958e26, State: Initialized, Role: FOLLOWER
I20250810 20:30:09.989349 5003 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc8ae5752c5467da2df9907c1958e26" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } }
I20250810 20:30:09.989840 5003 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:30:09.990083 5003 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:30:09.990374 5003 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:30:09.994868 5003 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc8ae5752c5467da2df9907c1958e26" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } }
I20250810 20:30:09.995683 5003 leader_election.cc:304] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dc8ae5752c5467da2df9907c1958e26; no voters:
I20250810 20:30:09.997484 5003 leader_election.cc:290] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:30:09.998368 5008 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:30:10.000591 5008 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [term 1 LEADER]: Becoming Leader. State: Replica: 0dc8ae5752c5467da2df9907c1958e26, State: Running, Role: LEADER
I20250810 20:30:10.001610 5003 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:30:10.001327 5008 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc8ae5752c5467da2df9907c1958e26" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } }
I20250810 20:30:10.013911 5010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dc8ae5752c5467da2df9907c1958e26. Latest consensus state: current_term: 1 leader_uuid: "0dc8ae5752c5467da2df9907c1958e26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc8ae5752c5467da2df9907c1958e26" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } } }
I20250810 20:30:10.013970 5009 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dc8ae5752c5467da2df9907c1958e26" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dc8ae5752c5467da2df9907c1958e26" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } } }
I20250810 20:30:10.014827 5009 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [sys.catalog]: This master's current role is: LEADER
I20250810 20:30:10.014716 5010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26 [sys.catalog]: This master's current role is: LEADER
I20250810 20:30:10.017469 5018 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:30:10.029515 5018 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:30:10.046727 5018 catalog_manager.cc:1349] Generated new cluster ID: 99b7ea823be040c7aaf2afcd863aad24
I20250810 20:30:10.047147 5018 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:30:10.076189 5018 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 20:30:10.077732 5018 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:30:10.093555 5018 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0dc8ae5752c5467da2df9907c1958e26: Generated new TSK 0
I20250810 20:30:10.094488 5018 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250810 20:30:10.109409 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:0
--local_ip_for_outbound_sockets=127.30.243.193
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:36065
--builtin_ntp_servers=127.30.243.212:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250810 20:30:10.441896 5027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:10.442449 5027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:30:10.443022 5027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:10.478049 5027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:30:10.479033 5027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:30:10.518546 5027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=0
--tserver_master_addrs=127.30.243.254:36065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:30:10.520077 5027 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:10.521840 5027 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:10.536425 5033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:11.937932 5032 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 5027
W20250810 20:30:12.482002 5027 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.945s user 0.738s sys 1.145s
W20250810 20:30:12.482645 5027 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.945s user 0.738s sys 1.145s
W20250810 20:30:10.538576 5034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:12.483943 5035 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1946 milliseconds
W20250810 20:30:12.485625 5036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:30:12.486145 5027 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:30:12.488703 5027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:30:12.491035 5027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:30:12.492475 5027 hybrid_clock.cc:648] HybridClock initialized: now 1754857812492407 us; error 64 us; skew 500 ppm
I20250810 20:30:12.493377 5027 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:12.500397 5027 webserver.cc:489] Webserver started at http://127.30.243.193:43067/ using document root <none> and password file <none>
I20250810 20:30:12.501448 5027 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:12.501688 5027 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:12.502158 5027 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:30:12.507004 5027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/instance:
uuid: "9676eaf6e3ef4c55a95abc56c8171f76"
format_stamp: "Formatted at 2025-08-10 20:30:12 on dist-test-slave-7swh"
I20250810 20:30:12.508189 5027 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal/instance:
uuid: "9676eaf6e3ef4c55a95abc56c8171f76"
format_stamp: "Formatted at 2025-08-10 20:30:12 on dist-test-slave-7swh"
I20250810 20:30:12.516220 5027 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250810 20:30:12.522622 5043 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:12.524340 5027 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250810 20:30:12.524828 5027 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal
uuid: "9676eaf6e3ef4c55a95abc56c8171f76"
format_stamp: "Formatted at 2025-08-10 20:30:12 on dist-test-slave-7swh"
I20250810 20:30:12.525210 5027 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:12.577037 5027 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:12.578519 5027 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:12.579047 5027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:12.582163 5027 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:30:12.586683 5027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:30:12.586984 5027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250810 20:30:12.587257 5027 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:30:12.587415 5027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:12.764807 5027 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:43069
I20250810 20:30:12.764928 5157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:43069 every 8 connection(s)
I20250810 20:30:12.767668 5027 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/info.pb
I20250810 20:30:12.771912 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 5027
I20250810 20:30:12.772461 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal/instance
I20250810 20:30:12.780668 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.194:0
--local_ip_for_outbound_sockets=127.30.243.194
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:36065
--builtin_ntp_servers=127.30.243.212:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250810 20:30:12.808511 5158 heartbeater.cc:344] Connected to a master server at 127.30.243.254:36065
I20250810 20:30:12.809008 5158 heartbeater.cc:461] Registering TS with master...
I20250810 20:30:12.810091 5158 heartbeater.cc:507] Master 127.30.243.254:36065 requested a full tablet report, sending...
I20250810 20:30:12.812851 4967 ts_manager.cc:194] Registered new tserver with Master: 9676eaf6e3ef4c55a95abc56c8171f76 (127.30.243.193:43069)
I20250810 20:30:12.814962 4967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:47881
W20250810 20:30:13.123454 5162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:13.123939 5162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:30:13.124390 5162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:13.157733 5162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:30:13.158574 5162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.194
I20250810 20:30:13.194876 5162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.243.194
--webserver_port=0
--tserver_master_addrs=127.30.243.254:36065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.194
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:30:13.196389 5162 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:13.198158 5162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:13.212244 5168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:30:13.818640 5158 heartbeater.cc:499] Master 127.30.243.254:36065 was elected leader, sending a full tablet report...
W20250810 20:30:13.212827 5169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:14.446193 5171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:14.449666 5162 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.237s user 0.427s sys 0.803s
W20250810 20:30:14.450074 5162 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.238s user 0.427s sys 0.803s
W20250810 20:30:14.450124 5170 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1235 milliseconds
I20250810 20:30:14.450371 5162 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:30:14.451597 5162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:30:14.453823 5162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:30:14.455195 5162 hybrid_clock.cc:648] HybridClock initialized: now 1754857814455154 us; error 72 us; skew 500 ppm
I20250810 20:30:14.456032 5162 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:14.463469 5162 webserver.cc:489] Webserver started at http://127.30.243.194:43609/ using document root <none> and password file <none>
I20250810 20:30:14.464603 5162 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:14.464874 5162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:14.465353 5162 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:30:14.470331 5162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/instance:
uuid: "a393ac6ab237410cb87735f1f92e4fdd"
format_stamp: "Formatted at 2025-08-10 20:30:14 on dist-test-slave-7swh"
I20250810 20:30:14.471654 5162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal/instance:
uuid: "a393ac6ab237410cb87735f1f92e4fdd"
format_stamp: "Formatted at 2025-08-10 20:30:14 on dist-test-slave-7swh"
I20250810 20:30:14.481580 5162 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.000s
I20250810 20:30:14.488302 5178 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:14.489681 5162 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20250810 20:30:14.490059 5162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal
uuid: "a393ac6ab237410cb87735f1f92e4fdd"
format_stamp: "Formatted at 2025-08-10 20:30:14 on dist-test-slave-7swh"
I20250810 20:30:14.490433 5162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:14.574808 5162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:14.576593 5162 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:14.577095 5162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:14.579838 5162 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:30:14.584218 5162 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:30:14.584456 5162 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:14.584710 5162 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:30:14.584868 5162 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:14.721189 5162 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.194:46155
I20250810 20:30:14.721330 5290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.194:46155 every 8 connection(s)
I20250810 20:30:14.723845 5162 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/info.pb
I20250810 20:30:14.724637 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 5162
I20250810 20:30:14.725176 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal/instance
I20250810 20:30:14.732872 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.195:0
--local_ip_for_outbound_sockets=127.30.243.195
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:36065
--builtin_ntp_servers=127.30.243.212:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250810 20:30:14.749580 5291 heartbeater.cc:344] Connected to a master server at 127.30.243.254:36065
I20250810 20:30:14.750372 5291 heartbeater.cc:461] Registering TS with master...
I20250810 20:30:14.751650 5291 heartbeater.cc:507] Master 127.30.243.254:36065 requested a full tablet report, sending...
I20250810 20:30:14.754383 4967 ts_manager.cc:194] Registered new tserver with Master: a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194:46155)
I20250810 20:30:14.756851 4967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.194:36613
W20250810 20:30:15.080705 5296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:15.081209 5296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:30:15.081795 5296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:15.116756 5296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:30:15.117715 5296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.195
I20250810 20:30:15.153714 5296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.243.195
--webserver_port=0
--tserver_master_addrs=127.30.243.254:36065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.195
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:30:15.155076 5296 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:15.156702 5296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:15.170770 5303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:30:15.761059 5291 heartbeater.cc:499] Master 127.30.243.254:36065 was elected leader, sending a full tablet report...
W20250810 20:30:15.169966 5302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:15.174953 5305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:16.398003 5304 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1224 milliseconds
I20250810 20:30:16.398133 5296 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:30:16.399444 5296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:30:16.402240 5296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:30:16.403697 5296 hybrid_clock.cc:648] HybridClock initialized: now 1754857816403647 us; error 64 us; skew 500 ppm
I20250810 20:30:16.404502 5296 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:16.410784 5296 webserver.cc:489] Webserver started at http://127.30.243.195:41501/ using document root <none> and password file <none>
I20250810 20:30:16.411818 5296 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:16.412067 5296 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:16.412550 5296 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:30:16.417235 5296 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/instance:
uuid: "2e1a6eeabb9741c6bdca8f441b80e942"
format_stamp: "Formatted at 2025-08-10 20:30:16 on dist-test-slave-7swh"
I20250810 20:30:16.418365 5296 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal/instance:
uuid: "2e1a6eeabb9741c6bdca8f441b80e942"
format_stamp: "Formatted at 2025-08-10 20:30:16 on dist-test-slave-7swh"
I20250810 20:30:16.425539 5296 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250810 20:30:16.431398 5312 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:16.432442 5296 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250810 20:30:16.432740 5296 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal
uuid: "2e1a6eeabb9741c6bdca8f441b80e942"
format_stamp: "Formatted at 2025-08-10 20:30:16 on dist-test-slave-7swh"
I20250810 20:30:16.433046 5296 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:16.491822 5296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:16.493260 5296 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:16.493667 5296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:16.496194 5296 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:30:16.500533 5296 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250810 20:30:16.500752 5296 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:16.500960 5296 ts_tablet_manager.cc:610] Registered 0 tablets
I20250810 20:30:16.501093 5296 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:16.643868 5296 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.195:37585
I20250810 20:30:16.643993 5424 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.195:37585 every 8 connection(s)
I20250810 20:30:16.646534 5296 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/info.pb
I20250810 20:30:16.655405 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 5296
I20250810 20:30:16.655834 31695 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal/instance
I20250810 20:30:16.673123 5425 heartbeater.cc:344] Connected to a master server at 127.30.243.254:36065
I20250810 20:30:16.673693 5425 heartbeater.cc:461] Registering TS with master...
I20250810 20:30:16.674855 5425 heartbeater.cc:507] Master 127.30.243.254:36065 requested a full tablet report, sending...
I20250810 20:30:16.677186 4967 ts_manager.cc:194] Registered new tserver with Master: 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585)
I20250810 20:30:16.678397 4967 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.195:47939
I20250810 20:30:16.691145 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:30:16.723485 31695 test_util.cc:276] Using random seed: -1395912700
I20250810 20:30:16.766489 4967 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60444:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250810 20:30:16.769186 4967 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 20:30:16.832248 5226 tablet_service.cc:1468] Processing CreateTablet for tablet 53ee6240b85240e4bc9fcfd330a40a0f (DEFAULT_TABLE table=pre_rebuild [id=a2126b0816a94811be1eb8bb393f6f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:30:16.833875 5226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53ee6240b85240e4bc9fcfd330a40a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:30:16.833231 5360 tablet_service.cc:1468] Processing CreateTablet for tablet 53ee6240b85240e4bc9fcfd330a40a0f (DEFAULT_TABLE table=pre_rebuild [id=a2126b0816a94811be1eb8bb393f6f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:30:16.833171 5091 tablet_service.cc:1468] Processing CreateTablet for tablet 53ee6240b85240e4bc9fcfd330a40a0f (DEFAULT_TABLE table=pre_rebuild [id=a2126b0816a94811be1eb8bb393f6f3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:30:16.835013 5360 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53ee6240b85240e4bc9fcfd330a40a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:30:16.836239 5091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53ee6240b85240e4bc9fcfd330a40a0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:30:16.859664 5450 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Bootstrap starting.
I20250810 20:30:16.864854 5451 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Bootstrap starting.
I20250810 20:30:16.868830 5452 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Bootstrap starting.
I20250810 20:30:16.870162 5450 tablet_bootstrap.cc:654] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Neither blocks nor log segments found. Creating new log.
I20250810 20:30:16.872920 5450 log.cc:826] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Log is configured to *not* fsync() on all Append() calls
I20250810 20:30:16.874797 5451 tablet_bootstrap.cc:654] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Neither blocks nor log segments found. Creating new log.
I20250810 20:30:16.876052 5452 tablet_bootstrap.cc:654] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Neither blocks nor log segments found. Creating new log.
I20250810 20:30:16.878528 5451 log.cc:826] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Log is configured to *not* fsync() on all Append() calls
I20250810 20:30:16.878793 5452 log.cc:826] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Log is configured to *not* fsync() on all Append() calls
I20250810 20:30:16.883466 5450 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: No bootstrap required, opened a new log
I20250810 20:30:16.884135 5450 ts_tablet_manager.cc:1397] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Time spent bootstrapping tablet: real 0.025s user 0.014s sys 0.004s
I20250810 20:30:16.887449 5451 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: No bootstrap required, opened a new log
I20250810 20:30:16.888269 5451 ts_tablet_manager.cc:1397] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.009s
I20250810 20:30:16.893962 5452 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: No bootstrap required, opened a new log
I20250810 20:30:16.894623 5452 ts_tablet_manager.cc:1397] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.014s
I20250810 20:30:16.916779 5451 raft_consensus.cc:357] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:16.923588 5452 raft_consensus.cc:357] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:16.924693 5452 raft_consensus.cc:383] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:30:16.925052 5452 raft_consensus.cc:738] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9676eaf6e3ef4c55a95abc56c8171f76, State: Initialized, Role: FOLLOWER
I20250810 20:30:16.924244 5450 raft_consensus.cc:357] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:16.926258 5452 consensus_queue.cc:260] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:16.926954 5450 raft_consensus.cc:383] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:30:16.926995 5451 raft_consensus.cc:383] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:30:16.927347 5451 raft_consensus.cc:738] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a393ac6ab237410cb87735f1f92e4fdd, State: Initialized, Role: FOLLOWER
I20250810 20:30:16.927413 5450 raft_consensus.cc:738] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1a6eeabb9741c6bdca8f441b80e942, State: Initialized, Role: FOLLOWER
I20250810 20:30:16.930531 5452 ts_tablet_manager.cc:1428] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Time spent starting tablet: real 0.036s user 0.022s sys 0.010s
I20250810 20:30:16.931883 5451 consensus_queue.cc:260] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:16.936149 5451 ts_tablet_manager.cc:1428] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Time spent starting tablet: real 0.048s user 0.031s sys 0.002s
I20250810 20:30:16.928720 5450 consensus_queue.cc:260] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:16.945444 5425 heartbeater.cc:499] Master 127.30.243.254:36065 was elected leader, sending a full tablet report...
I20250810 20:30:16.949448 5450 ts_tablet_manager.cc:1428] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Time spent starting tablet: real 0.065s user 0.031s sys 0.008s
I20250810 20:30:16.980370 5457 raft_consensus.cc:491] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:30:16.980960 5457 raft_consensus.cc:513] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
W20250810 20:30:16.981823 5292 tablet.cc:2378] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:30:16.984028 5457 leader_election.cc:290] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585), 9676eaf6e3ef4c55a95abc56c8171f76 (127.30.243.193:43069)
I20250810 20:30:17.000772 5380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ee6240b85240e4bc9fcfd330a40a0f" candidate_uuid: "a393ac6ab237410cb87735f1f92e4fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" is_pre_election: true
I20250810 20:30:17.001797 5380 raft_consensus.cc:2466] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a393ac6ab237410cb87735f1f92e4fdd in term 0.
I20250810 20:30:17.003408 5182 leader_election.cc:304] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1a6eeabb9741c6bdca8f441b80e942, a393ac6ab237410cb87735f1f92e4fdd; no voters:
I20250810 20:30:17.004668 5457 raft_consensus.cc:2802] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 20:30:17.005065 5457 raft_consensus.cc:491] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 20:30:17.005437 5457 raft_consensus.cc:3058] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:30:17.006100 5111 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ee6240b85240e4bc9fcfd330a40a0f" candidate_uuid: "a393ac6ab237410cb87735f1f92e4fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" is_pre_election: true
I20250810 20:30:17.007117 5111 raft_consensus.cc:2466] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a393ac6ab237410cb87735f1f92e4fdd in term 0.
I20250810 20:30:17.010315 5457 raft_consensus.cc:513] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:17.011739 5457 leader_election.cc:290] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [CANDIDATE]: Term 1 election: Requested vote from peers 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585), 9676eaf6e3ef4c55a95abc56c8171f76 (127.30.243.193:43069)
I20250810 20:30:17.012787 5380 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ee6240b85240e4bc9fcfd330a40a0f" candidate_uuid: "a393ac6ab237410cb87735f1f92e4fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2e1a6eeabb9741c6bdca8f441b80e942"
I20250810 20:30:17.013082 5111 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ee6240b85240e4bc9fcfd330a40a0f" candidate_uuid: "a393ac6ab237410cb87735f1f92e4fdd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9676eaf6e3ef4c55a95abc56c8171f76"
I20250810 20:30:17.013410 5380 raft_consensus.cc:3058] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:30:17.013540 5111 raft_consensus.cc:3058] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:30:17.018013 5111 raft_consensus.cc:2466] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a393ac6ab237410cb87735f1f92e4fdd in term 1.
I20250810 20:30:17.018989 5182 leader_election.cc:304] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9676eaf6e3ef4c55a95abc56c8171f76, a393ac6ab237410cb87735f1f92e4fdd; no voters:
I20250810 20:30:17.019631 5457 raft_consensus.cc:2802] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:30:17.021730 5380 raft_consensus.cc:2466] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a393ac6ab237410cb87735f1f92e4fdd in term 1.
I20250810 20:30:17.022872 5457 raft_consensus.cc:695] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 1 LEADER]: Becoming Leader. State: Replica: a393ac6ab237410cb87735f1f92e4fdd, State: Running, Role: LEADER
I20250810 20:30:17.024210 5457 consensus_queue.cc:237] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
W20250810 20:30:17.028384 5159 tablet.cc:2378] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:30:17.036504 4967 catalog_manager.cc:5582] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd reported cstate change: term changed from 0 to 1, leader changed from <none> to a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194). New cstate: current_term: 1 leader_uuid: "a393ac6ab237410cb87735f1f92e4fdd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } health_report { overall_health: UNKNOWN } } }
W20250810 20:30:17.152310 5426 tablet.cc:2378] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:30:17.239317 5380 raft_consensus.cc:1273] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 1 FOLLOWER]: Refusing update from remote peer a393ac6ab237410cb87735f1f92e4fdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250810 20:30:17.239387 5111 raft_consensus.cc:1273] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Refusing update from remote peer a393ac6ab237410cb87735f1f92e4fdd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250810 20:30:17.240845 5457 consensus_queue.cc:1035] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250810 20:30:17.241894 5461 consensus_queue.cc:1035] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [LEADER]: Connected to new peer: Peer: permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 20:30:17.272754 5469 mvcc.cc:204] Tried to move back new op lower bound from 7187897619394514944 to 7187897618539044864. Current Snapshot: MvccSnapshot[applied={T|T < 7187897619394514944}]
I20250810 20:30:22.693099 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 4933
W20250810 20:30:23.169909 5504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:23.170547 5504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:23.203917 5504 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250810 20:30:23.347395 5291 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:36065 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:36065: connect: Connection refused (error 111)
W20250810 20:30:23.349316 5425 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:36065 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:36065: connect: Connection refused (error 111)
W20250810 20:30:23.383924 5158 heartbeater.cc:646] Failed to heartbeat to 127.30.243.254:36065 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.30.243.254:36065: connect: Connection refused (error 111)
W20250810 20:30:24.653613 5510 debug-util.cc:398] Leaking SignalData structure 0x7b08000373e0 after lost signal to thread 5504
W20250810 20:30:24.654186 5510 kernel_stack_watchdog.cc:198] Thread 5504 stuck at /home/jenkins-slave/workspace/build_and_test_flaky/src/kudu/util/thread.cc:642 for 403ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250810 20:30:24.834534 5504 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.587s user 0.642s sys 0.912s
W20250810 20:30:24.939723 5504 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.692s user 0.650s sys 0.920s
I20250810 20:30:25.026527 5504 minidump.cc:252] Setting minidump size limit to 20M
I20250810 20:30:25.029081 5504 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:25.030408 5504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:25.041841 5540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:25.042351 5541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:25.046445 5543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:30:25.046753 5504 server_base.cc:1047] running on GCE node
I20250810 20:30:25.048063 5504 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250810 20:30:25.048627 5504 hybrid_clock.cc:648] HybridClock initialized: now 1754857825048589 us; error 142327 us; skew 500 ppm
I20250810 20:30:25.049381 5504 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:25.055119 5504 webserver.cc:489] Webserver started at http://0.0.0.0:43653/ using document root <none> and password file <none>
I20250810 20:30:25.056314 5504 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:25.056592 5504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:25.057164 5504 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250810 20:30:25.061875 5504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/instance:
uuid: "5e497d2f74e644f78fbd48e049dcef62"
format_stamp: "Formatted at 2025-08-10 20:30:25 on dist-test-slave-7swh"
I20250810 20:30:25.063900 5504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal/instance:
uuid: "5e497d2f74e644f78fbd48e049dcef62"
format_stamp: "Formatted at 2025-08-10 20:30:25 on dist-test-slave-7swh"
I20250810 20:30:25.070735 5504 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250810 20:30:25.076076 5548 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:25.077114 5504 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250810 20:30:25.077577 5504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
uuid: "5e497d2f74e644f78fbd48e049dcef62"
format_stamp: "Formatted at 2025-08-10 20:30:25 on dist-test-slave-7swh"
I20250810 20:30:25.078070 5504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:25.320583 5504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:25.322101 5504 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:25.322535 5504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:25.327312 5504 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:30:25.342167 5504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: Bootstrap starting.
I20250810 20:30:25.347174 5504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: Neither blocks nor log segments found. Creating new log.
I20250810 20:30:25.348845 5504 log.cc:826] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: Log is configured to *not* fsync() on all Append() calls
I20250810 20:30:25.353118 5504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: No bootstrap required, opened a new log
I20250810 20:30:25.370164 5504 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER }
I20250810 20:30:25.370664 5504 raft_consensus.cc:383] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:30:25.370851 5504 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e497d2f74e644f78fbd48e049dcef62, State: Initialized, Role: FOLLOWER
I20250810 20:30:25.371539 5504 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER }
I20250810 20:30:25.371989 5504 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:30:25.372208 5504 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:30:25.372475 5504 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:30:25.376552 5504 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER }
I20250810 20:30:25.377384 5504 leader_election.cc:304] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e497d2f74e644f78fbd48e049dcef62; no voters:
I20250810 20:30:25.379405 5504 leader_election.cc:290] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250810 20:30:25.379644 5557 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:30:25.383669 5557 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 1 LEADER]: Becoming Leader. State: Replica: 5e497d2f74e644f78fbd48e049dcef62, State: Running, Role: LEADER
I20250810 20:30:25.384909 5557 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER }
I20250810 20:30:25.394654 5558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e497d2f74e644f78fbd48e049dcef62. Latest consensus state: current_term: 1 leader_uuid: "5e497d2f74e644f78fbd48e049dcef62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER } }
I20250810 20:30:25.395220 5558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [sys.catalog]: This master's current role is: LEADER
I20250810 20:30:25.395678 5560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e497d2f74e644f78fbd48e049dcef62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER } }
I20250810 20:30:25.396097 5560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [sys.catalog]: This master's current role is: LEADER
I20250810 20:30:25.407809 5504 tablet_replica.cc:331] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: stopping tablet replica
I20250810 20:30:25.408380 5504 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 1 LEADER]: Raft consensus shutting down.
I20250810 20:30:25.408803 5504 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250810 20:30:25.410981 5504 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250810 20:30:25.411434 5504 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250810 20:30:25.470248 5504 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250810 20:30:26.502171 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 5027
I20250810 20:30:26.540812 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 5162
I20250810 20:30:26.589835 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 5296
I20250810 20:30:26.628083 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:36065
--webserver_interface=127.30.243.254
--webserver_port=40573
--builtin_ntp_servers=127.30.243.212:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.30.243.254:36065 with env {}
W20250810 20:30:26.966200 5569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:26.966908 5569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:30:26.967687 5569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:27.002895 5569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250810 20:30:27.003325 5569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:30:27.003582 5569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250810 20:30:27.003810 5569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250810 20:30:27.043458 5569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.30.243.254:36065
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.30.243.254:36065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.30.243.254
--webserver_port=40573
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:30:27.044816 5569 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:27.046490 5569 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:27.058009 5575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:27.059458 5576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:27.068634 5578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:28.263721 5577 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250810 20:30:28.263871 5569 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:30:28.267963 5569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:30:28.271448 5569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:30:28.272869 5569 hybrid_clock.cc:648] HybridClock initialized: now 1754857828272817 us; error 56 us; skew 500 ppm
I20250810 20:30:28.273751 5569 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:28.280514 5569 webserver.cc:489] Webserver started at http://127.30.243.254:40573/ using document root <none> and password file <none>
I20250810 20:30:28.281558 5569 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:28.281811 5569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:28.290403 5569 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.004s
I20250810 20:30:28.295387 5585 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:28.296662 5569 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20250810 20:30:28.297046 5569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
uuid: "5e497d2f74e644f78fbd48e049dcef62"
format_stamp: "Formatted at 2025-08-10 20:30:25 on dist-test-slave-7swh"
I20250810 20:30:28.299155 5569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:28.361678 5569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:28.363305 5569 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:28.363783 5569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:28.439852 5569 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.254:36065
I20250810 20:30:28.440057 5636 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.254:36065 every 8 connection(s)
I20250810 20:30:28.442852 5569 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/info.pb
I20250810 20:30:28.447135 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 5569
I20250810 20:30:28.449390 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.193:43069
--local_ip_for_outbound_sockets=127.30.243.193
--tserver_master_addrs=127.30.243.254:36065
--webserver_port=43067
--webserver_interface=127.30.243.193
--builtin_ntp_servers=127.30.243.212:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250810 20:30:28.458588 5637 sys_catalog.cc:263] Verifying existing consensus state
I20250810 20:30:28.473383 5637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: Bootstrap starting.
I20250810 20:30:28.483991 5637 log.cc:826] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: Log is configured to *not* fsync() on all Append() calls
I20250810 20:30:28.496659 5637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:30:28.497481 5637 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: Bootstrap complete.
I20250810 20:30:28.519901 5637 raft_consensus.cc:357] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } }
I20250810 20:30:28.520637 5637 raft_consensus.cc:738] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e497d2f74e644f78fbd48e049dcef62, State: Initialized, Role: FOLLOWER
I20250810 20:30:28.521492 5637 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } }
I20250810 20:30:28.522027 5637 raft_consensus.cc:397] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250810 20:30:28.522315 5637 raft_consensus.cc:491] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250810 20:30:28.522634 5637 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:30:28.526988 5637 raft_consensus.cc:513] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } }
I20250810 20:30:28.527714 5637 leader_election.cc:304] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e497d2f74e644f78fbd48e049dcef62; no voters:
I20250810 20:30:28.530169 5637 leader_election.cc:290] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250810 20:30:28.530766 5641 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 20:30:28.534036 5641 raft_consensus.cc:695] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [term 2 LEADER]: Becoming Leader. State: Replica: 5e497d2f74e644f78fbd48e049dcef62, State: Running, Role: LEADER
I20250810 20:30:28.535043 5641 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } }
I20250810 20:30:28.535816 5637 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [sys.catalog]: configured and running, proceeding with master startup.
I20250810 20:30:28.546427 5642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5e497d2f74e644f78fbd48e049dcef62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } } }
I20250810 20:30:28.548537 5642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [sys.catalog]: This master's current role is: LEADER
I20250810 20:30:28.547784 5643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e497d2f74e644f78fbd48e049dcef62. Latest consensus state: current_term: 2 leader_uuid: "5e497d2f74e644f78fbd48e049dcef62" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e497d2f74e644f78fbd48e049dcef62" member_type: VOTER last_known_addr { host: "127.30.243.254" port: 36065 } } }
I20250810 20:30:28.550256 5643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62 [sys.catalog]: This master's current role is: LEADER
I20250810 20:30:28.553805 5649 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250810 20:30:28.566534 5649 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=c5ae22f333d442a4a5eb9025887d06e6]
I20250810 20:30:28.574349 5649 tablet_loader.cc:96] loaded metadata for tablet 53ee6240b85240e4bc9fcfd330a40a0f (table pre_rebuild [id=c5ae22f333d442a4a5eb9025887d06e6])
I20250810 20:30:28.576125 5649 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250810 20:30:28.607483 5649 catalog_manager.cc:1349] Generated new cluster ID: c01f2321562848679e534e4d289a8318
I20250810 20:30:28.607924 5649 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250810 20:30:28.643476 5660 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20250810 20:30:28.667296 5649 catalog_manager.cc:1372] Generated new certificate authority record
I20250810 20:30:28.668869 5649 catalog_manager.cc:1506] Loading token signing keys...
I20250810 20:30:28.683353 5649 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: Generated new TSK 0
I20250810 20:30:28.684295 5649 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250810 20:30:28.856817 5639 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:28.857390 5639 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:30:28.857972 5639 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:28.891876 5639 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:30:28.892794 5639 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.193
I20250810 20:30:28.932758 5639 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.193:43069
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.30.243.193
--webserver_port=43067
--tserver_master_addrs=127.30.243.254:36065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.193
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:30:28.934156 5639 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:28.936017 5639 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:28.950147 5666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:28.954866 5669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:28.952649 5667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:30.500870 5668 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1545 milliseconds
I20250810 20:30:30.501020 5639 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:30:30.502903 5639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:30:30.505956 5639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:30:30.507460 5639 hybrid_clock.cc:648] HybridClock initialized: now 1754857830507387 us; error 84 us; skew 500 ppm
I20250810 20:30:30.508323 5639 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:30.515591 5639 webserver.cc:489] Webserver started at http://127.30.243.193:43067/ using document root <none> and password file <none>
I20250810 20:30:30.516635 5639 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:30.516871 5639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:30.526517 5639 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20250810 20:30:30.532066 5677 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:30.533275 5639 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250810 20:30:30.533618 5639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal
uuid: "9676eaf6e3ef4c55a95abc56c8171f76"
format_stamp: "Formatted at 2025-08-10 20:30:12 on dist-test-slave-7swh"
I20250810 20:30:30.535701 5639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:30.595186 5639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:30.596753 5639 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:30.597272 5639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:30.600953 5639 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:30:30.608088 5684 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 20:30:30.616647 5639 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 20:30:30.616899 5639 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.000s sys 0.003s
I20250810 20:30:30.617228 5639 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 20:30:30.622459 5639 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 20:30:30.622753 5639 ts_tablet_manager.cc:589] Time spent register tablets: real 0.006s user 0.002s sys 0.000s
I20250810 20:30:30.623232 5684 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Bootstrap starting.
I20250810 20:30:30.847216 5639 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.193:43069
I20250810 20:30:30.847429 5790 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.193:43069 every 8 connection(s)
I20250810 20:30:30.851408 5639 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/info.pb
I20250810 20:30:30.852784 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 5639
I20250810 20:30:30.855252 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.194:46155
--local_ip_for_outbound_sockets=127.30.243.194
--tserver_master_addrs=127.30.243.254:36065
--webserver_port=43609
--webserver_interface=127.30.243.194
--builtin_ntp_servers=127.30.243.212:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250810 20:30:30.924578 5791 heartbeater.cc:344] Connected to a master server at 127.30.243.254:36065
I20250810 20:30:30.925127 5791 heartbeater.cc:461] Registering TS with master...
I20250810 20:30:30.926501 5791 heartbeater.cc:507] Master 127.30.243.254:36065 requested a full tablet report, sending...
I20250810 20:30:30.931875 5602 ts_manager.cc:194] Registered new tserver with Master: 9676eaf6e3ef4c55a95abc56c8171f76 (127.30.243.193:43069)
I20250810 20:30:30.940335 5602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.193:42441
I20250810 20:30:30.981316 5684 log.cc:826] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Log is configured to *not* fsync() on all Append() calls
W20250810 20:30:31.373713 5792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:31.374724 5792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:30:31.375592 5792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:31.436573 5792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:30:31.438113 5792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.194
I20250810 20:30:31.506613 5792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.194:46155
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.30.243.194
--webserver_port=43609
--tserver_master_addrs=127.30.243.254:36065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.194
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:30:31.508379 5792 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:31.510746 5792 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:31.529839 5802 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:30:31.945330 5791 heartbeater.cc:499] Master 127.30.243.254:36065 was elected leader, sending a full tablet report...
W20250810 20:30:32.929772 5801 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 5792
W20250810 20:30:33.238173 5792 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.711s user 0.630s sys 1.053s
W20250810 20:30:31.529937 5803 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:33.240466 5792 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.714s user 0.630s sys 1.053s
I20250810 20:30:33.247689 5792 server_base.cc:1047] running on GCE node
W20250810 20:30:33.249348 5807 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:30:33.250865 5792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:30:33.253623 5792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:30:33.255101 5792 hybrid_clock.cc:648] HybridClock initialized: now 1754857833255045 us; error 57 us; skew 500 ppm
I20250810 20:30:33.256189 5792 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:33.264576 5792 webserver.cc:489] Webserver started at http://127.30.243.194:43609/ using document root <none> and password file <none>
I20250810 20:30:33.266077 5792 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:33.266455 5792 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:33.279075 5792 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.006s sys 0.001s
I20250810 20:30:33.285754 5812 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:33.287127 5792 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250810 20:30:33.287535 5792 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal
uuid: "a393ac6ab237410cb87735f1f92e4fdd"
format_stamp: "Formatted at 2025-08-10 20:30:14 on dist-test-slave-7swh"
I20250810 20:30:33.290439 5792 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:33.352933 5684 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:30:33.353919 5684 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Bootstrap complete.
I20250810 20:30:33.355707 5684 ts_tablet_manager.cc:1397] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Time spent bootstrapping tablet: real 2.733s user 2.634s sys 0.077s
I20250810 20:30:33.373189 5792 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:33.371877 5684 raft_consensus.cc:357] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:33.375121 5684 raft_consensus.cc:738] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9676eaf6e3ef4c55a95abc56c8171f76, State: Initialized, Role: FOLLOWER
I20250810 20:30:33.375355 5792 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:33.375902 5792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:33.376065 5684 consensus_queue.cc:260] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:33.380107 5684 ts_tablet_manager.cc:1428] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Time spent starting tablet: real 0.024s user 0.023s sys 0.001s
I20250810 20:30:33.380290 5792 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:30:33.387480 5820 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 20:30:33.395606 5792 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 20:30:33.395848 5792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.000s sys 0.003s
I20250810 20:30:33.396152 5792 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 20:30:33.400892 5792 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 20:30:33.401270 5792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.000s sys 0.005s
I20250810 20:30:33.401589 5820 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Bootstrap starting.
I20250810 20:30:33.585492 5792 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.194:46155
I20250810 20:30:33.585851 5926 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.194:46155 every 8 connection(s)
I20250810 20:30:33.588454 5792 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/info.pb
I20250810 20:30:33.596771 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 5792
I20250810 20:30:33.598805 31695 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
/tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.30.243.195:37585
--local_ip_for_outbound_sockets=127.30.243.195
--tserver_master_addrs=127.30.243.254:36065
--webserver_port=41501
--webserver_interface=127.30.243.195
--builtin_ntp_servers=127.30.243.212:42767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250810 20:30:33.615358 5927 heartbeater.cc:344] Connected to a master server at 127.30.243.254:36065
I20250810 20:30:33.615880 5927 heartbeater.cc:461] Registering TS with master...
I20250810 20:30:33.617141 5927 heartbeater.cc:507] Master 127.30.243.254:36065 requested a full tablet report, sending...
I20250810 20:30:33.621444 5602 ts_manager.cc:194] Registered new tserver with Master: a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194:46155)
I20250810 20:30:33.624883 5602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.194:50835
I20250810 20:30:33.684183 5820 log.cc:826] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Log is configured to *not* fsync() on all Append() calls
W20250810 20:30:33.939533 5931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250810 20:30:33.940071 5931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250810 20:30:33.940583 5931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250810 20:30:33.974404 5931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250810 20:30:33.975373 5931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.30.243.195
I20250810 20:30:34.012449 5931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.30.243.212:42767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.30.243.195:37585
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.30.243.195
--webserver_port=41501
--tserver_master_addrs=127.30.243.254:36065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.30.243.195
--log_dir=/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision b92f16d1c86a753c597b46c7575bfa6a1479726a
build type FASTDEBUG
built by None at 10 Aug 2025 20:25:32 UTC on 24a791456cd2
build id 7505
TSAN enabled
I20250810 20:30:34.013892 5931 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250810 20:30:34.015642 5931 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250810 20:30:34.029910 5938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250810 20:30:34.628918 5927 heartbeater.cc:499] Master 127.30.243.254:36065 was elected leader, sending a full tablet report...
I20250810 20:30:35.020377 5944 raft_consensus.cc:491] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:30:35.031652 5944 raft_consensus.cc:513] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
W20250810 20:30:35.042111 5681 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.30.243.195:37585: connect: Connection refused (error 111)
I20250810 20:30:35.041205 5944 leader_election.cc:290] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585), a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194:46155)
W20250810 20:30:35.077880 5681 leader_election.cc:336] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585): Network error: Client connection negotiation failed: client connection to 127.30.243.195:37585: connect: Connection refused (error 111)
I20250810 20:30:35.081797 5882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ee6240b85240e4bc9fcfd330a40a0f" candidate_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "a393ac6ab237410cb87735f1f92e4fdd" is_pre_election: true
W20250810 20:30:35.110849 5678 leader_election.cc:343] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194:46155): Illegal state: must be running to vote when last-logged opid is not known
I20250810 20:30:35.111596 5678 leader_election.cc:304] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9676eaf6e3ef4c55a95abc56c8171f76; no voters: 2e1a6eeabb9741c6bdca8f441b80e942, a393ac6ab237410cb87735f1f92e4fdd
I20250810 20:30:35.112854 5944 raft_consensus.cc:2747] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250810 20:30:34.031150 5939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:35.431311 5937 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 5931
W20250810 20:30:35.762425 5931 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.732s user 0.577s sys 1.124s
W20250810 20:30:35.764642 5931 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.735s user 0.577s sys 1.125s
W20250810 20:30:35.765209 5941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250810 20:30:35.770035 5940 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1733 milliseconds
I20250810 20:30:35.770140 5931 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250810 20:30:35.771525 5931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250810 20:30:35.774040 5931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250810 20:30:35.775422 5931 hybrid_clock.cc:648] HybridClock initialized: now 1754857835775367 us; error 73 us; skew 500 ppm
I20250810 20:30:35.776232 5931 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250810 20:30:35.782727 5931 webserver.cc:489] Webserver started at http://127.30.243.195:41501/ using document root <none> and password file <none>
I20250810 20:30:35.783844 5931 fs_manager.cc:362] Metadata directory not provided
I20250810 20:30:35.784123 5931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250810 20:30:35.792761 5931 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20250810 20:30:35.798043 5953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250810 20:30:35.799579 5931 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20250810 20:30:35.799963 5931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data,/tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal
uuid: "2e1a6eeabb9741c6bdca8f441b80e942"
format_stamp: "Formatted at 2025-08-10 20:30:16 on dist-test-slave-7swh"
I20250810 20:30:35.801980 5931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250810 20:30:35.854826 5931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250810 20:30:35.856488 5931 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250810 20:30:35.857033 5931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250810 20:30:35.861146 5931 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250810 20:30:35.868026 5960 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250810 20:30:35.876361 5931 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250810 20:30:35.876613 5931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.003s sys 0.000s
I20250810 20:30:35.876929 5931 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250810 20:30:35.881793 5931 ts_tablet_manager.cc:610] Registered 1 tablets
I20250810 20:30:35.882128 5931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.006s sys 0.000s
I20250810 20:30:35.882534 5960 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Bootstrap starting.
I20250810 20:30:36.081933 5931 rpc_server.cc:307] RPC server started. Bound to: 127.30.243.195:37585
I20250810 20:30:36.082129 6066 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.30.243.195:37585 every 8 connection(s)
I20250810 20:30:36.085932 5931 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/info.pb
I20250810 20:30:36.095566 31695 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu as pid 5931
I20250810 20:30:36.184811 6067 heartbeater.cc:344] Connected to a master server at 127.30.243.254:36065
I20250810 20:30:36.185395 6067 heartbeater.cc:461] Registering TS with master...
I20250810 20:30:36.186843 6067 heartbeater.cc:507] Master 127.30.243.254:36065 requested a full tablet report, sending...
I20250810 20:30:36.191996 5602 ts_manager.cc:194] Registered new tserver with Master: 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585)
I20250810 20:30:36.196511 5602 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.30.243.195:57035
I20250810 20:30:36.198985 31695 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250810 20:30:36.226284 5960 log.cc:826] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Log is configured to *not* fsync() on all Append() calls
I20250810 20:30:36.909569 5820 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:30:36.910399 5820 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Bootstrap complete.
I20250810 20:30:36.911909 5820 ts_tablet_manager.cc:1397] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Time spent bootstrapping tablet: real 3.511s user 3.407s sys 0.057s
I20250810 20:30:36.917507 5820 raft_consensus.cc:357] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:36.919698 5820 raft_consensus.cc:738] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a393ac6ab237410cb87735f1f92e4fdd, State: Initialized, Role: FOLLOWER
I20250810 20:30:36.920435 5820 consensus_queue.cc:260] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:36.923655 5820 ts_tablet_manager.cc:1428] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20250810 20:30:37.203095 6067 heartbeater.cc:499] Master 127.30.243.254:36065 was elected leader, sending a full tablet report...
I20250810 20:30:37.317581 6081 raft_consensus.cc:491] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:30:37.318006 6081 raft_consensus.cc:513] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:37.319530 6081 leader_election.cc:290] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585), a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194:46155)
I20250810 20:30:37.320794 5882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ee6240b85240e4bc9fcfd330a40a0f" candidate_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "a393ac6ab237410cb87735f1f92e4fdd" is_pre_election: true
I20250810 20:30:37.321672 5882 raft_consensus.cc:2466] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9676eaf6e3ef4c55a95abc56c8171f76 in term 1.
I20250810 20:30:37.323509 5678 leader_election.cc:304] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9676eaf6e3ef4c55a95abc56c8171f76, a393ac6ab237410cb87735f1f92e4fdd; no voters:
I20250810 20:30:37.325814 6081 raft_consensus.cc:2802] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250810 20:30:37.326270 6081 raft_consensus.cc:491] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 20:30:37.326686 6081 raft_consensus.cc:3058] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:30:37.334751 6081 raft_consensus.cc:513] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:37.337358 6081 leader_election.cc:290] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [CANDIDATE]: Term 2 election: Requested vote from peers 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585), a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194:46155)
I20250810 20:30:37.338403 5882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ee6240b85240e4bc9fcfd330a40a0f" candidate_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "a393ac6ab237410cb87735f1f92e4fdd"
I20250810 20:30:37.338869 5882 raft_consensus.cc:3058] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 1 FOLLOWER]: Advancing to term 2
I20250810 20:30:37.348891 5882 raft_consensus.cc:2466] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9676eaf6e3ef4c55a95abc56c8171f76 in term 2.
I20250810 20:30:37.350394 5678 leader_election.cc:304] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9676eaf6e3ef4c55a95abc56c8171f76, a393ac6ab237410cb87735f1f92e4fdd; no voters:
I20250810 20:30:37.351357 6081 raft_consensus.cc:2802] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 2 FOLLOWER]: Leader election won for term 2
I20250810 20:30:37.339983 6022 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ee6240b85240e4bc9fcfd330a40a0f" candidate_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" is_pre_election: true
I20250810 20:30:37.340421 6021 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ee6240b85240e4bc9fcfd330a40a0f" candidate_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "2e1a6eeabb9741c6bdca8f441b80e942"
W20250810 20:30:37.354045 5681 leader_election.cc:343] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585): Illegal state: must be running to vote when last-logged opid is not known
I20250810 20:30:37.354493 6081 raft_consensus.cc:695] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 2 LEADER]: Becoming Leader. State: Replica: 9676eaf6e3ef4c55a95abc56c8171f76, State: Running, Role: LEADER
I20250810 20:30:37.355640 6081 consensus_queue.cc:237] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:37.367759 5602 catalog_manager.cc:5582] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 reported cstate change: term changed from 0 to 2, leader changed from <none> to 9676eaf6e3ef4c55a95abc56c8171f76 (127.30.243.193), VOTER 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195) added, VOTER 9676eaf6e3ef4c55a95abc56c8171f76 (127.30.243.193) added, VOTER a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194) added. New cstate: current_term: 2 leader_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } health_report { overall_health: HEALTHY } } }
I20250810 20:30:37.854404 5882 raft_consensus.cc:1273] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 2 FOLLOWER]: Refusing update from remote peer 9676eaf6e3ef4c55a95abc56c8171f76: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250810 20:30:37.856781 6081 consensus_queue.cc:1035] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250810 20:30:37.924701 5681 consensus_peers.cc:489] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 -> Peer 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585): Couldn't send request to peer 2e1a6eeabb9741c6bdca8f441b80e942. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250810 20:30:37.953512 5746 consensus_queue.cc:237] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:37.959548 5882 raft_consensus.cc:1273] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 2 FOLLOWER]: Refusing update from remote peer 9676eaf6e3ef4c55a95abc56c8171f76: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250810 20:30:37.961282 6096 consensus_queue.cc:1035] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250810 20:30:37.969300 6096 raft_consensus.cc:2953] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } } }
I20250810 20:30:37.972246 5882 raft_consensus.cc:2953] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } } }
I20250810 20:30:37.986459 5589 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 53ee6240b85240e4bc9fcfd330a40a0f with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250810 20:30:37.989321 5601 catalog_manager.cc:5582] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd reported cstate change: config changed from index -1 to 207, VOTER 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195) evicted. New cstate: current_term: 2 leader_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } } }
I20250810 20:30:38.035660 5746 consensus_queue.cc:237] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:38.038654 6081 raft_consensus.cc:2953] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } } }
I20250810 20:30:38.050468 5589 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 53ee6240b85240e4bc9fcfd330a40a0f with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250810 20:30:38.054363 5602 catalog_manager.cc:5582] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 reported cstate change: config changed from index 207 to 208, VOTER a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194) evicted. New cstate: current_term: 2 leader_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } health_report { overall_health: HEALTHY } } }
I20250810 20:30:38.070294 6002 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ee6240b85240e4bc9fcfd330a40a0f with delete_type TABLET_DATA_TOMBSTONED (TS 2e1a6eeabb9741c6bdca8f441b80e942 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:59052
W20250810 20:30:38.093484 5589 catalog_manager.cc:4908] TS 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585): delete failed for tablet 53ee6240b85240e4bc9fcfd330a40a0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53ee6240b85240e4bc9fcfd330a40a0f already in progress: opening tablet
I20250810 20:30:38.094388 5862 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ee6240b85240e4bc9fcfd330a40a0f with delete_type TABLET_DATA_TOMBSTONED (TS a393ac6ab237410cb87735f1f92e4fdd not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:55622
I20250810 20:30:38.110188 6106 tablet_replica.cc:331] stopping tablet replica
I20250810 20:30:38.111286 6106 raft_consensus.cc:2241] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 2 FOLLOWER]: Raft consensus shutting down.
I20250810 20:30:38.112031 6106 raft_consensus.cc:2270] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd [term 2 FOLLOWER]: Raft consensus is shut down!
I20250810 20:30:38.142505 6106 ts_tablet_manager.cc:1905] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250810 20:30:38.160737 6106 ts_tablet_manager.cc:1918] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250810 20:30:38.161338 6106 log.cc:1199] T 53ee6240b85240e4bc9fcfd330a40a0f P a393ac6ab237410cb87735f1f92e4fdd: Deleting WAL directory at /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/wal/wals/53ee6240b85240e4bc9fcfd330a40a0f
I20250810 20:30:38.163400 5586 catalog_manager.cc:4928] TS a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194:46155): tablet 53ee6240b85240e4bc9fcfd330a40a0f (table pre_rebuild [id=c5ae22f333d442a4a5eb9025887d06e6]) successfully deleted
W20250810 20:30:38.209192 31695 scanner-internal.cc:458] Time spent opening tablet: real 1.969s user 0.008s sys 0.000s
I20250810 20:30:38.649019 5960 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250810 20:30:38.650292 5960 tablet_bootstrap.cc:492] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Bootstrap complete.
I20250810 20:30:38.652278 5960 ts_tablet_manager.cc:1397] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Time spent bootstrapping tablet: real 2.770s user 2.659s sys 0.061s
I20250810 20:30:38.661180 5960 raft_consensus.cc:357] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:38.664255 5960 raft_consensus.cc:738] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1a6eeabb9741c6bdca8f441b80e942, State: Initialized, Role: FOLLOWER
I20250810 20:30:38.665227 5960 consensus_queue.cc:260] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:38.672281 5960 ts_tablet_manager.cc:1428] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Time spent starting tablet: real 0.020s user 0.017s sys 0.000s
I20250810 20:30:38.676936 6002 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ee6240b85240e4bc9fcfd330a40a0f with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:59052
I20250810 20:30:38.683338 6115 tablet_replica.cc:331] stopping tablet replica
I20250810 20:30:38.684187 6115 raft_consensus.cc:2241] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250810 20:30:38.684768 6115 raft_consensus.cc:2270] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250810 20:30:38.712512 6115 ts_tablet_manager.cc:1905] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250810 20:30:38.732087 6115 ts_tablet_manager.cc:1918] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250810 20:30:38.732537 6115 log.cc:1199] T 53ee6240b85240e4bc9fcfd330a40a0f P 2e1a6eeabb9741c6bdca8f441b80e942: Deleting WAL directory at /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/wal/wals/53ee6240b85240e4bc9fcfd330a40a0f
I20250810 20:30:38.734264 5589 catalog_manager.cc:4928] TS 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195:37585): tablet 53ee6240b85240e4bc9fcfd330a40a0f (table pre_rebuild [id=c5ae22f333d442a4a5eb9025887d06e6]) successfully deleted
I20250810 20:30:38.979112 6002 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 20:30:38.995364 5726 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 20:30:38.997557 5862 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
5e497d2f74e644f78fbd48e049dcef62 | 127.30.243.254:36065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.30.243.212:42767 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
2e1a6eeabb9741c6bdca8f441b80e942 | 127.30.243.195:37585 | HEALTHY | <none> | 0 | 0
9676eaf6e3ef4c55a95abc56c8171f76 | 127.30.243.193:43069 | HEALTHY | <none> | 1 | 0
a393ac6ab237410cb87735f1f92e4fdd | 127.30.243.194:46155 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.30.243.193 | experimental | 127.30.243.193:43069
local_ip_for_outbound_sockets | 127.30.243.194 | experimental | 127.30.243.194:46155
local_ip_for_outbound_sockets | 127.30.243.195 | experimental | 127.30.243.195:37585
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/info.pb | hidden | 127.30.243.193:43069
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/info.pb | hidden | 127.30.243.194:46155
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/info.pb | hidden | 127.30.243.195:37585
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.30.243.212:42767 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250810 20:30:39.315332 31695 log_verifier.cc:126] Checking tablet 53ee6240b85240e4bc9fcfd330a40a0f
I20250810 20:30:39.657665 31695 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 53ee6240b85240e4bc9fcfd330a40a0f
I20250810 20:30:39.660984 5601 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42858:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250810 20:30:39.661633 5601 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42858:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250810 20:30:39.676419 5601 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 5e497d2f74e644f78fbd48e049dcef62: Sending DeleteTablet for 1 replicas of tablet 53ee6240b85240e4bc9fcfd330a40a0f
I20250810 20:30:39.678500 5726 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ee6240b85240e4bc9fcfd330a40a0f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-10 20:30:39 UTC) from {username='slave'} at 127.0.0.1:42422
I20250810 20:30:39.678911 31695 test_util.cc:276] Using random seed: -1372957279
I20250810 20:30:39.680756 6142 tablet_replica.cc:331] stopping tablet replica
I20250810 20:30:39.681619 6142 raft_consensus.cc:2241] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 2 LEADER]: Raft consensus shutting down.
I20250810 20:30:39.682435 6142 raft_consensus.cc:2270] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250810 20:30:39.719132 5601 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42892:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250810 20:30:39.723021 5601 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250810 20:30:39.734447 6142 ts_tablet_manager.cc:1905] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250810 20:30:39.752365 6142 ts_tablet_manager.cc:1918] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250810 20:30:39.752990 6142 log.cc:1199] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Deleting WAL directory at /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/wal/wals/53ee6240b85240e4bc9fcfd330a40a0f
I20250810 20:30:39.754151 6142 ts_tablet_manager.cc:1939] T 53ee6240b85240e4bc9fcfd330a40a0f P 9676eaf6e3ef4c55a95abc56c8171f76: Deleting consensus metadata
I20250810 20:30:39.754170 6002 tablet_service.cc:1468] Processing CreateTablet for tablet 678a34f47ad7459fabdd5e061dce99ce (DEFAULT_TABLE table=post_rebuild [id=4edacc983af74b09a9dd9d994c39294e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:30:39.755667 5862 tablet_service.cc:1468] Processing CreateTablet for tablet 678a34f47ad7459fabdd5e061dce99ce (DEFAULT_TABLE table=post_rebuild [id=4edacc983af74b09a9dd9d994c39294e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:30:39.757090 5862 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678a34f47ad7459fabdd5e061dce99ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:30:39.757035 5726 tablet_service.cc:1468] Processing CreateTablet for tablet 678a34f47ad7459fabdd5e061dce99ce (DEFAULT_TABLE table=post_rebuild [id=4edacc983af74b09a9dd9d994c39294e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250810 20:30:39.757855 5589 catalog_manager.cc:4928] TS 9676eaf6e3ef4c55a95abc56c8171f76 (127.30.243.193:43069): tablet 53ee6240b85240e4bc9fcfd330a40a0f (table pre_rebuild [id=c5ae22f333d442a4a5eb9025887d06e6]) successfully deleted
I20250810 20:30:39.758538 5726 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678a34f47ad7459fabdd5e061dce99ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:30:39.759555 6002 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 678a34f47ad7459fabdd5e061dce99ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250810 20:30:39.780210 6150 tablet_bootstrap.cc:492] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942: Bootstrap starting.
I20250810 20:30:39.787410 6149 tablet_bootstrap.cc:492] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd: Bootstrap starting.
I20250810 20:30:39.791103 6151 tablet_bootstrap.cc:492] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76: Bootstrap starting.
I20250810 20:30:39.792192 6150 tablet_bootstrap.cc:654] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942: Neither blocks nor log segments found. Creating new log.
I20250810 20:30:39.792781 6149 tablet_bootstrap.cc:654] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd: Neither blocks nor log segments found. Creating new log.
I20250810 20:30:39.802306 6149 tablet_bootstrap.cc:492] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd: No bootstrap required, opened a new log
I20250810 20:30:39.803088 6149 ts_tablet_manager.cc:1397] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd: Time spent bootstrapping tablet: real 0.016s user 0.010s sys 0.004s
I20250810 20:30:39.804857 6151 tablet_bootstrap.cc:654] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76: Neither blocks nor log segments found. Creating new log.
I20250810 20:30:39.806901 6149 raft_consensus.cc:357] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:39.808732 6149 raft_consensus.cc:383] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:30:39.809145 6149 raft_consensus.cc:738] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a393ac6ab237410cb87735f1f92e4fdd, State: Initialized, Role: FOLLOWER
I20250810 20:30:39.809994 6149 consensus_queue.cc:260] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:39.810850 6150 tablet_bootstrap.cc:492] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942: No bootstrap required, opened a new log
I20250810 20:30:39.811398 6150 ts_tablet_manager.cc:1397] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.020s
I20250810 20:30:39.816731 6150 raft_consensus.cc:357] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:39.817768 6150 raft_consensus.cc:383] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:30:39.818135 6150 raft_consensus.cc:738] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e1a6eeabb9741c6bdca8f441b80e942, State: Initialized, Role: FOLLOWER
I20250810 20:30:39.819003 6150 consensus_queue.cc:260] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:39.819506 6149 ts_tablet_manager.cc:1428] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd: Time spent starting tablet: real 0.016s user 0.006s sys 0.008s
I20250810 20:30:39.819824 6151 tablet_bootstrap.cc:492] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76: No bootstrap required, opened a new log
I20250810 20:30:39.820353 6151 ts_tablet_manager.cc:1397] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.003s
I20250810 20:30:39.823597 6151 raft_consensus.cc:357] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:39.826025 6151 raft_consensus.cc:383] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250810 20:30:39.826398 6151 raft_consensus.cc:738] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9676eaf6e3ef4c55a95abc56c8171f76, State: Initialized, Role: FOLLOWER
I20250810 20:30:39.827319 6151 consensus_queue.cc:260] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:39.834705 6150 ts_tablet_manager.cc:1428] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942: Time spent starting tablet: real 0.023s user 0.016s sys 0.003s
I20250810 20:30:39.835722 6151 ts_tablet_manager.cc:1428] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76: Time spent starting tablet: real 0.015s user 0.011s sys 0.000s
W20250810 20:30:39.858891 5928 tablet.cc:2378] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 20:30:39.914963 5793 tablet.cc:2378] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250810 20:30:39.921520 6068 tablet.cc:2378] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250810 20:30:40.025662 6156 raft_consensus.cc:491] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250810 20:30:40.026328 6156 raft_consensus.cc:513] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:40.029587 6156 leader_election.cc:290] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194:46155), 9676eaf6e3ef4c55a95abc56c8171f76 (127.30.243.193:43069)
I20250810 20:30:40.043751 5882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "678a34f47ad7459fabdd5e061dce99ce" candidate_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a393ac6ab237410cb87735f1f92e4fdd" is_pre_election: true
I20250810 20:30:40.044298 5882 raft_consensus.cc:2466] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e1a6eeabb9741c6bdca8f441b80e942 in term 0.
I20250810 20:30:40.046214 5954 leader_election.cc:304] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1a6eeabb9741c6bdca8f441b80e942, a393ac6ab237410cb87735f1f92e4fdd; no voters:
I20250810 20:30:40.046926 6156 raft_consensus.cc:2802] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250810 20:30:40.047325 6156 raft_consensus.cc:491] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250810 20:30:40.047752 6156 raft_consensus.cc:3058] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:30:40.048523 5746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "678a34f47ad7459fabdd5e061dce99ce" candidate_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" is_pre_election: true
I20250810 20:30:40.049440 5746 raft_consensus.cc:2466] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2e1a6eeabb9741c6bdca8f441b80e942 in term 0.
I20250810 20:30:40.054363 6156 raft_consensus.cc:513] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:40.056459 6156 leader_election.cc:290] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [CANDIDATE]: Term 1 election: Requested vote from peers a393ac6ab237410cb87735f1f92e4fdd (127.30.243.194:46155), 9676eaf6e3ef4c55a95abc56c8171f76 (127.30.243.193:43069)
I20250810 20:30:40.057531 5882 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "678a34f47ad7459fabdd5e061dce99ce" candidate_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a393ac6ab237410cb87735f1f92e4fdd"
I20250810 20:30:40.057680 5746 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "678a34f47ad7459fabdd5e061dce99ce" candidate_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9676eaf6e3ef4c55a95abc56c8171f76"
I20250810 20:30:40.058115 5882 raft_consensus.cc:3058] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:30:40.058255 5746 raft_consensus.cc:3058] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76 [term 0 FOLLOWER]: Advancing to term 1
I20250810 20:30:40.062707 5882 raft_consensus.cc:2466] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e1a6eeabb9741c6bdca8f441b80e942 in term 1.
I20250810 20:30:40.063874 5954 leader_election.cc:304] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2e1a6eeabb9741c6bdca8f441b80e942, a393ac6ab237410cb87735f1f92e4fdd; no voters:
I20250810 20:30:40.064525 6156 raft_consensus.cc:2802] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 1 FOLLOWER]: Leader election won for term 1
I20250810 20:30:40.065249 5746 raft_consensus.cc:2466] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2e1a6eeabb9741c6bdca8f441b80e942 in term 1.
I20250810 20:30:40.068342 6156 raft_consensus.cc:695] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [term 1 LEADER]: Becoming Leader. State: Replica: 2e1a6eeabb9741c6bdca8f441b80e942, State: Running, Role: LEADER
I20250810 20:30:40.069921 6156 consensus_queue.cc:237] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } }
I20250810 20:30:40.083717 5601 catalog_manager.cc:5582] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2e1a6eeabb9741c6bdca8f441b80e942 (127.30.243.195). New cstate: current_term: 1 leader_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2e1a6eeabb9741c6bdca8f441b80e942" member_type: VOTER last_known_addr { host: "127.30.243.195" port: 37585 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 } health_report { overall_health: UNKNOWN } } }
I20250810 20:30:40.261511 5746 raft_consensus.cc:1273] T 678a34f47ad7459fabdd5e061dce99ce P 9676eaf6e3ef4c55a95abc56c8171f76 [term 1 FOLLOWER]: Refusing update from remote peer 2e1a6eeabb9741c6bdca8f441b80e942: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250810 20:30:40.263711 6162 consensus_queue.cc:1035] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9676eaf6e3ef4c55a95abc56c8171f76" member_type: VOTER last_known_addr { host: "127.30.243.193" port: 43069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250810 20:30:40.264906 5882 raft_consensus.cc:1273] T 678a34f47ad7459fabdd5e061dce99ce P a393ac6ab237410cb87735f1f92e4fdd [term 1 FOLLOWER]: Refusing update from remote peer 2e1a6eeabb9741c6bdca8f441b80e942: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250810 20:30:40.266489 6162 consensus_queue.cc:1035] T 678a34f47ad7459fabdd5e061dce99ce P 2e1a6eeabb9741c6bdca8f441b80e942 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a393ac6ab237410cb87735f1f92e4fdd" member_type: VOTER last_known_addr { host: "127.30.243.194" port: 46155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250810 20:30:40.302793 6171 mvcc.cc:204] Tried to move back new op lower bound from 7187897713692200960 to 7187897712936808448. Current Snapshot: MvccSnapshot[applied={T|T < 7187897713692200960}]
I20250810 20:30:45.548532 6002 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250810 20:30:45.554080 5726 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250810 20:30:45.554418 5862 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
5e497d2f74e644f78fbd48e049dcef62 | 127.30.243.254:36065 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.30.243.212:42767 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
2e1a6eeabb9741c6bdca8f441b80e942 | 127.30.243.195:37585 | HEALTHY | <none> | 1 | 0
9676eaf6e3ef4c55a95abc56c8171f76 | 127.30.243.193:43069 | HEALTHY | <none> | 0 | 0
a393ac6ab237410cb87735f1f92e4fdd | 127.30.243.194:46155 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.30.243.193 | experimental | 127.30.243.193:43069
local_ip_for_outbound_sockets | 127.30.243.194 | experimental | 127.30.243.194:46155
local_ip_for_outbound_sockets | 127.30.243.195 | experimental | 127.30.243.195:37585
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-0/data/info.pb | hidden | 127.30.243.193:43069
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-1/data/info.pb | hidden | 127.30.243.194:46155
server_dump_info_path | /tmp/dist-test-taskWjaPOb/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1754857648852986-31695-0/minicluster-data/ts-2/data/info.pb | hidden | 127.30.243.195:37585
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.30.243.212:42767 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250810 20:30:45.786991 31695 log_verifier.cc:126] Checking tablet 53ee6240b85240e4bc9fcfd330a40a0f
I20250810 20:30:45.787464 31695 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 53ee6240b85240e4bc9fcfd330a40a0f
I20250810 20:30:45.787672 31695 log_verifier.cc:126] Checking tablet 678a34f47ad7459fabdd5e061dce99ce
I20250810 20:30:46.588209 31695 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 678a34f47ad7459fabdd5e061dce99ce
I20250810 20:30:46.615577 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 5639
I20250810 20:30:46.666405 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 5792
W20250810 20:30:46.705582 5954 connection.cc:537] client connection to 127.30.243.194:46155 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250810 20:30:46.706017 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 5931
I20250810 20:30:46.751576 31695 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskWjaPOb/build/tsan/bin/kudu with pid 5569
2025-08-10T20:30:46Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (38802 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (38802 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (197893 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST