Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250814 07:57:10.295838 20960 test_util.cc:276] Using random seed: -1630051166
W20250814 07:57:11.478003 20960 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.139s user 0.450s sys 0.688s
W20250814 07:57:11.478366 20960 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.139s user 0.450s sys 0.688s
I20250814 07:57:11.493755 20960 ts_itest-base.cc:115] Starting cluster with:
I20250814 07:57:11.493920 20960 ts_itest-base.cc:116] --------------
I20250814 07:57:11.494086 20960 ts_itest-base.cc:117] 4 tablet servers
I20250814 07:57:11.494244 20960 ts_itest-base.cc:118] 3 replicas per TS
I20250814 07:57:11.494395 20960 ts_itest-base.cc:119] --------------
2025-08-14T07:57:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T07:57:11Z Disabled control of system clock
I20250814 07:57:11.527773 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:44207
--webserver_interface=127.20.120.62
--webserver_port=0
--builtin_ntp_servers=127.20.120.20:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:44207 with env {}
W20250814 07:57:11.809099 20974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:11.809581 20974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:11.809959 20974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:11.838979 20974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:57:11.839215 20974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:11.839406 20974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:57:11.839596 20974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:57:11.872751 20974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:44207
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:11.874037 20974 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:11.875495 20974 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:11.886400 20980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:11.887353 20981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:13.028426 20983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:13.031622 20982 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1142 milliseconds
I20250814 07:57:13.031702 20974 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:13.032924 20974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:13.035557 20974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:13.036914 20974 hybrid_clock.cc:648] HybridClock initialized: now 1755158233036868 us; error 45 us; skew 500 ppm
I20250814 07:57:13.037883 20974 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:13.044147 20974 webserver.cc:480] Webserver started at http://127.20.120.62:37455/ using document root <none> and password file <none>
I20250814 07:57:13.045099 20974 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:13.045295 20974 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:13.045737 20974 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:13.050719 20974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851"
format_stamp: "Formatted at 2025-08-14 07:57:13 on dist-test-slave-87c1"
I20250814 07:57:13.051698 20974 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851"
format_stamp: "Formatted at 2025-08-14 07:57:13 on dist-test-slave-87c1"
I20250814 07:57:13.058604 20974 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20250814 07:57:13.063843 20990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:13.064956 20974 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250814 07:57:13.065327 20974 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851"
format_stamp: "Formatted at 2025-08-14 07:57:13 on dist-test-slave-87c1"
I20250814 07:57:13.065642 20974 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:13.119868 20974 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:13.121203 20974 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:13.121630 20974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:13.188838 20974 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:44207
I20250814 07:57:13.188907 21041 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:44207 every 8 connection(s)
I20250814 07:57:13.191519 20974 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 07:57:13.193221 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 20974
I20250814 07:57:13.193625 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 07:57:13.197504 21042 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:13.217424 21042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851: Bootstrap starting.
I20250814 07:57:13.222726 21042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:13.224520 21042 log.cc:826] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:13.229130 21042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851: No bootstrap required, opened a new log
I20250814 07:57:13.246246 21042 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } }
I20250814 07:57:13.246837 21042 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:13.247133 21042 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb5c4c525bc442db9a5f0aa0cbf4851, State: Initialized, Role: FOLLOWER
I20250814 07:57:13.247874 21042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } }
I20250814 07:57:13.248370 21042 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:57:13.248633 21042 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:57:13.248940 21042 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:13.252781 21042 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } }
I20250814 07:57:13.253582 21042 leader_election.cc:304] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bb5c4c525bc442db9a5f0aa0cbf4851; no voters:
I20250814 07:57:13.255154 21042 leader_election.cc:290] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:57:13.255910 21047 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:57:13.258013 21047 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 1 LEADER]: Becoming Leader. State: Replica: 9bb5c4c525bc442db9a5f0aa0cbf4851, State: Running, Role: LEADER
I20250814 07:57:13.258625 21047 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } }
I20250814 07:57:13.258996 21042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:57:13.268617 21048 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } } }
I20250814 07:57:13.270087 21048 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [sys.catalog]: This master's current role is: LEADER
I20250814 07:57:13.270174 21049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bb5c4c525bc442db9a5f0aa0cbf4851. Latest consensus state: current_term: 1 leader_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } } }
I20250814 07:57:13.271739 21049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [sys.catalog]: This master's current role is: LEADER
I20250814 07:57:13.273824 21056 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:57:13.286315 21056 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:57:13.301656 21056 catalog_manager.cc:1349] Generated new cluster ID: 77bfc86e448947e5934392fd1e8f2216
I20250814 07:57:13.301862 21056 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:57:13.320611 21056 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 07:57:13.321926 21056 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:57:13.339732 21056 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851: Generated new TSK 0
I20250814 07:57:13.340415 21056 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 07:57:13.361912 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:0
--local_ip_for_outbound_sockets=127.20.120.1
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:44207
--builtin_ntp_servers=127.20.120.20:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250814 07:57:13.643484 21066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:13.643952 21066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:13.644454 21066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:13.673769 21066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:13.674541 21066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 07:57:13.707338 21066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:44207
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:13.708590 21066 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:13.710148 21066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:13.722409 21072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:15.126029 21071 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 21066
W20250814 07:57:13.723980 21073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:15.412551 21066 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.689s user 0.545s sys 1.096s
W20250814 07:57:15.414616 21075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:15.414799 21066 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.691s user 0.545s sys 1.097s
W20250814 07:57:15.414609 21074 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1690 milliseconds
I20250814 07:57:15.415180 21066 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:15.419560 21066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:15.421700 21066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:15.423041 21066 hybrid_clock.cc:648] HybridClock initialized: now 1755158235423007 us; error 35 us; skew 500 ppm
I20250814 07:57:15.423812 21066 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:15.430027 21066 webserver.cc:480] Webserver started at http://127.20.120.1:35273/ using document root <none> and password file <none>
I20250814 07:57:15.430867 21066 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:15.431098 21066 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:15.431532 21066 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:15.435775 21066 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "78537f0a126b4fc99a3f75df886f6638"
format_stamp: "Formatted at 2025-08-14 07:57:15 on dist-test-slave-87c1"
I20250814 07:57:15.436836 21066 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "78537f0a126b4fc99a3f75df886f6638"
format_stamp: "Formatted at 2025-08-14 07:57:15 on dist-test-slave-87c1"
I20250814 07:57:15.443974 21066 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250814 07:57:15.449397 21082 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:15.450424 21066 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250814 07:57:15.450701 21066 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "78537f0a126b4fc99a3f75df886f6638"
format_stamp: "Formatted at 2025-08-14 07:57:15 on dist-test-slave-87c1"
I20250814 07:57:15.450999 21066 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:15.498993 21066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:15.500389 21066 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:15.500809 21066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:15.503858 21066 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:57:15.508535 21066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:57:15.508755 21066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:15.509032 21066 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:57:15.509217 21066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20250814 07:57:15.664829 21066 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:45271
I20250814 07:57:15.664894 21194 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:45271 every 8 connection(s)
I20250814 07:57:15.668169 21066 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 07:57:15.670542 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 21066
I20250814 07:57:15.670998 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 07:57:15.678694 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.2:0
--local_ip_for_outbound_sockets=127.20.120.2
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:44207
--builtin_ntp_servers=127.20.120.20:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:57:15.711387 21195 heartbeater.cc:344] Connected to a master server at 127.20.120.62:44207
I20250814 07:57:15.711813 21195 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:15.713022 21195 heartbeater.cc:507] Master 127.20.120.62:44207 requested a full tablet report, sending...
I20250814 07:57:15.715776 21007 ts_manager.cc:194] Registered new tserver with Master: 78537f0a126b4fc99a3f75df886f6638 (127.20.120.1:45271)
I20250814 07:57:15.718361 21007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:45617
W20250814 07:57:15.984511 21199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:15.984954 21199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:15.985466 21199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:16.015357 21199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:16.016534 21199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.2
I20250814 07:57:16.049176 21199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:44207
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.2
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:16.050415 21199 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:16.051874 21199 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:16.063033 21205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:16.722025 21195 heartbeater.cc:499] Master 127.20.120.62:44207 was elected leader, sending a full tablet report...
W20250814 07:57:16.063633 21206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:17.297479 21207 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1229 milliseconds
W20250814 07:57:17.298063 21199 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.235s user 0.418s sys 0.801s
W20250814 07:57:17.298431 21199 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.235s user 0.418s sys 0.801s
W20250814 07:57:17.298857 21208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:17.298913 21199 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:17.300136 21199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:17.302335 21199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:17.303678 21199 hybrid_clock.cc:648] HybridClock initialized: now 1755158237303622 us; error 54 us; skew 500 ppm
I20250814 07:57:17.304446 21199 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:17.311743 21199 webserver.cc:480] Webserver started at http://127.20.120.2:45973/ using document root <none> and password file <none>
I20250814 07:57:17.312697 21199 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:17.312904 21199 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:17.313364 21199 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:17.317698 21199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1ae448f18b4b49bbb52063c2315a9fb2"
format_stamp: "Formatted at 2025-08-14 07:57:17 on dist-test-slave-87c1"
I20250814 07:57:17.318739 21199 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1ae448f18b4b49bbb52063c2315a9fb2"
format_stamp: "Formatted at 2025-08-14 07:57:17 on dist-test-slave-87c1"
I20250814 07:57:17.326089 21199 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20250814 07:57:17.331820 21215 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:17.332865 21199 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20250814 07:57:17.333200 21199 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1ae448f18b4b49bbb52063c2315a9fb2"
format_stamp: "Formatted at 2025-08-14 07:57:17 on dist-test-slave-87c1"
I20250814 07:57:17.333503 21199 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:17.393538 21199 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:17.394865 21199 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:17.395288 21199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:17.398337 21199 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:57:17.402426 21199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:57:17.402614 21199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:17.402864 21199 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:57:17.403007 21199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:17.532604 21199 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.2:41969
I20250814 07:57:17.532704 21327 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.2:41969 every 8 connection(s)
I20250814 07:57:17.535063 21199 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 07:57:17.539521 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 21199
I20250814 07:57:17.540043 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 07:57:17.545589 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.3:0
--local_ip_for_outbound_sockets=127.20.120.3
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:44207
--builtin_ntp_servers=127.20.120.20:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:57:17.555336 21328 heartbeater.cc:344] Connected to a master server at 127.20.120.62:44207
I20250814 07:57:17.555831 21328 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:17.557037 21328 heartbeater.cc:507] Master 127.20.120.62:44207 requested a full tablet report, sending...
I20250814 07:57:17.559176 21007 ts_manager.cc:194] Registered new tserver with Master: 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2:41969)
I20250814 07:57:17.560359 21007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.2:47315
W20250814 07:57:17.826234 21332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:17.826673 21332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:17.827152 21332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:17.856158 21332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:17.856966 21332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.3
I20250814 07:57:17.888099 21332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:44207
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.3
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:17.889300 21332 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:17.890733 21332 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:17.902088 21338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:18.563812 21328 heartbeater.cc:499] Master 127.20.120.62:44207 was elected leader, sending a full tablet report...
W20250814 07:57:19.305770 21337 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 21332
W20250814 07:57:19.606132 21332 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.704s user 0.650s sys 1.052s
W20250814 07:57:19.606559 21332 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.704s user 0.650s sys 1.052s
W20250814 07:57:17.902729 21339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:19.609028 21341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:19.612879 21340 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1703 milliseconds
I20250814 07:57:19.612896 21332 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:19.614327 21332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:19.617234 21332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:19.618638 21332 hybrid_clock.cc:648] HybridClock initialized: now 1755158239618555 us; error 73 us; skew 500 ppm
I20250814 07:57:19.619740 21332 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:19.627844 21332 webserver.cc:480] Webserver started at http://127.20.120.3:41291/ using document root <none> and password file <none>
I20250814 07:57:19.629284 21332 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:19.629616 21332 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:19.630257 21332 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:19.638067 21332 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "59025a646f504bcdb79badce6765ca6e"
format_stamp: "Formatted at 2025-08-14 07:57:19 on dist-test-slave-87c1"
I20250814 07:57:19.639816 21332 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "59025a646f504bcdb79badce6765ca6e"
format_stamp: "Formatted at 2025-08-14 07:57:19 on dist-test-slave-87c1"
I20250814 07:57:19.647064 21332 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.007s
I20250814 07:57:19.652377 21348 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:19.653395 21332 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250814 07:57:19.653702 21332 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "59025a646f504bcdb79badce6765ca6e"
format_stamp: "Formatted at 2025-08-14 07:57:19 on dist-test-slave-87c1"
I20250814 07:57:19.654002 21332 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:19.703505 21332 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:19.704881 21332 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:19.705324 21332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:19.707719 21332 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:57:19.711678 21332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:57:19.711870 21332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:19.712152 21332 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:57:19.712298 21332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:19.844789 21332 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.3:46493
I20250814 07:57:19.844884 21460 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.3:46493 every 8 connection(s)
I20250814 07:57:19.847307 21332 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 07:57:19.853302 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 21332
I20250814 07:57:19.853807 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 07:57:19.860317 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.4:0
--local_ip_for_outbound_sockets=127.20.120.4
--webserver_interface=127.20.120.4
--webserver_port=0
--tserver_master_addrs=127.20.120.62:44207
--builtin_ntp_servers=127.20.120.20:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:57:19.869107 21461 heartbeater.cc:344] Connected to a master server at 127.20.120.62:44207
I20250814 07:57:19.869482 21461 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:19.870446 21461 heartbeater.cc:507] Master 127.20.120.62:44207 requested a full tablet report, sending...
I20250814 07:57:19.872602 21007 ts_manager.cc:194] Registered new tserver with Master: 59025a646f504bcdb79badce6765ca6e (127.20.120.3:46493)
I20250814 07:57:19.874341 21007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.3:43243
W20250814 07:57:20.151289 21465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:20.151712 21465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:20.152180 21465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:20.181089 21465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:20.181792 21465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.4
I20250814 07:57:20.214407 21465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.120.4
--webserver_port=0
--tserver_master_addrs=127.20.120.62:44207
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.4
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:20.215503 21465 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:20.216931 21465 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:20.227377 21471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:20.877660 21461 heartbeater.cc:499] Master 127.20.120.62:44207 was elected leader, sending a full tablet report...
W20250814 07:57:20.229329 21472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:21.438629 21473 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1208 milliseconds
W20250814 07:57:21.439296 21474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:21.439324 21465 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:21.442279 21465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:21.445369 21465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:21.446695 21465 hybrid_clock.cc:648] HybridClock initialized: now 1755158241446648 us; error 45 us; skew 500 ppm
I20250814 07:57:21.447444 21465 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:21.454041 21465 webserver.cc:480] Webserver started at http://127.20.120.4:40597/ using document root <none> and password file <none>
I20250814 07:57:21.454898 21465 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:21.455096 21465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:21.455526 21465 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:21.459597 21465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "8748301262c74e6eb010c9e299822208"
format_stamp: "Formatted at 2025-08-14 07:57:21 on dist-test-slave-87c1"
I20250814 07:57:21.460633 21465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "8748301262c74e6eb010c9e299822208"
format_stamp: "Formatted at 2025-08-14 07:57:21 on dist-test-slave-87c1"
I20250814 07:57:21.467084 21465 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250814 07:57:21.472126 21482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:21.473047 21465 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250814 07:57:21.473344 21465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "8748301262c74e6eb010c9e299822208"
format_stamp: "Formatted at 2025-08-14 07:57:21 on dist-test-slave-87c1"
I20250814 07:57:21.473636 21465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:21.538239 21465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:21.539659 21465 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:21.540055 21465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:21.542356 21465 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:57:21.546114 21465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:57:21.546305 21465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:21.546525 21465 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:57:21.546672 21465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:21.679749 21465 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.4:35251
I20250814 07:57:21.679934 21594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.4:35251 every 8 connection(s)
I20250814 07:57:21.682334 21465 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250814 07:57:21.687426 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 21465
I20250814 07:57:21.688247 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250814 07:57:21.705699 21595 heartbeater.cc:344] Connected to a master server at 127.20.120.62:44207
I20250814 07:57:21.706080 21595 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:21.707228 21595 heartbeater.cc:507] Master 127.20.120.62:44207 requested a full tablet report, sending...
I20250814 07:57:21.709170 21007 ts_manager.cc:194] Registered new tserver with Master: 8748301262c74e6eb010c9e299822208 (127.20.120.4:35251)
I20250814 07:57:21.710351 21007 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.4:48081
I20250814 07:57:21.721796 20960 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250814 07:57:21.758582 21007 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33666:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250814 07:57:21.852658 21130 tablet_service.cc:1468] Processing CreateTablet for tablet 07f9fb6555f4487890d8b6f2b409b59f (DEFAULT_TABLE table=TestTable [id=873a82748fad4da3a524bc6cc70f63de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:57:21.855057 21130 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f9fb6555f4487890d8b6f2b409b59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:21.856545 21396 tablet_service.cc:1468] Processing CreateTablet for tablet 07f9fb6555f4487890d8b6f2b409b59f (DEFAULT_TABLE table=TestTable [id=873a82748fad4da3a524bc6cc70f63de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:57:21.856940 21263 tablet_service.cc:1468] Processing CreateTablet for tablet 07f9fb6555f4487890d8b6f2b409b59f (DEFAULT_TABLE table=TestTable [id=873a82748fad4da3a524bc6cc70f63de]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:57:21.858029 21396 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f9fb6555f4487890d8b6f2b409b59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:21.858624 21263 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f9fb6555f4487890d8b6f2b409b59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:21.883158 21615 tablet_bootstrap.cc:492] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638: Bootstrap starting.
I20250814 07:57:21.890249 21616 tablet_bootstrap.cc:492] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2: Bootstrap starting.
I20250814 07:57:21.891198 21615 tablet_bootstrap.cc:654] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:21.893803 21615 log.cc:826] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:21.895892 21616 tablet_bootstrap.cc:654] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:21.896011 21617 tablet_bootstrap.cc:492] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e: Bootstrap starting.
I20250814 07:57:21.898319 21616 log.cc:826] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:21.902877 21615 tablet_bootstrap.cc:492] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638: No bootstrap required, opened a new log
I20250814 07:57:21.903278 21615 ts_tablet_manager.cc:1397] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638: Time spent bootstrapping tablet: real 0.021s user 0.010s sys 0.010s
I20250814 07:57:21.903288 21617 tablet_bootstrap.cc:654] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:21.905341 21617 log.cc:826] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:21.906147 21616 tablet_bootstrap.cc:492] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2: No bootstrap required, opened a new log
I20250814 07:57:21.906651 21616 ts_tablet_manager.cc:1397] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20250814 07:57:21.909588 21617 tablet_bootstrap.cc:492] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e: No bootstrap required, opened a new log
I20250814 07:57:21.910288 21617 ts_tablet_manager.cc:1397] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.008s
I20250814 07:57:21.926954 21617 raft_consensus.cc:357] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:21.927556 21617 raft_consensus.cc:383] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:21.927790 21617 raft_consensus.cc:738] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59025a646f504bcdb79badce6765ca6e, State: Initialized, Role: FOLLOWER
I20250814 07:57:21.928520 21617 consensus_queue.cc:260] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:21.930538 21615 raft_consensus.cc:357] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:21.931443 21615 raft_consensus.cc:383] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:21.931787 21615 raft_consensus.cc:738] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78537f0a126b4fc99a3f75df886f6638, State: Initialized, Role: FOLLOWER
I20250814 07:57:21.932543 21617 ts_tablet_manager.cc:1428] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e: Time spent starting tablet: real 0.022s user 0.022s sys 0.000s
I20250814 07:57:21.932785 21615 consensus_queue.cc:260] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:21.933504 21616 raft_consensus.cc:357] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:21.934442 21616 raft_consensus.cc:383] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:21.934808 21616 raft_consensus.cc:738] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ae448f18b4b49bbb52063c2315a9fb2, State: Initialized, Role: FOLLOWER
I20250814 07:57:21.935837 21616 consensus_queue.cc:260] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:21.936592 21615 ts_tablet_manager.cc:1428] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638: Time spent starting tablet: real 0.033s user 0.029s sys 0.003s
I20250814 07:57:21.940029 21616 ts_tablet_manager.cc:1428] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2: Time spent starting tablet: real 0.033s user 0.029s sys 0.002s
W20250814 07:57:21.953612 21196 tablet.cc:2378] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:57:22.031301 21623 raft_consensus.cc:491] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:57:22.031762 21623 raft_consensus.cc:513] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:22.034610 21623 leader_election.cc:290] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 59025a646f504bcdb79badce6765ca6e (127.20.120.3:46493), 78537f0a126b4fc99a3f75df886f6638 (127.20.120.1:45271)
W20250814 07:57:22.043759 21329 tablet.cc:2378] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:57:22.046204 21416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07f9fb6555f4487890d8b6f2b409b59f" candidate_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59025a646f504bcdb79badce6765ca6e" is_pre_election: true
I20250814 07:57:22.046357 21150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07f9fb6555f4487890d8b6f2b409b59f" candidate_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78537f0a126b4fc99a3f75df886f6638" is_pre_election: true
I20250814 07:57:22.047138 21150 raft_consensus.cc:2466] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae448f18b4b49bbb52063c2315a9fb2 in term 0.
I20250814 07:57:22.047125 21416 raft_consensus.cc:2466] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1ae448f18b4b49bbb52063c2315a9fb2 in term 0.
I20250814 07:57:22.048339 21217 leader_election.cc:304] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae448f18b4b49bbb52063c2315a9fb2, 78537f0a126b4fc99a3f75df886f6638; no voters:
I20250814 07:57:22.049062 21623 raft_consensus.cc:2802] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 07:57:22.049361 21623 raft_consensus.cc:491] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 07:57:22.049566 21623 raft_consensus.cc:3058] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:22.053767 21623 raft_consensus.cc:513] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:22.055117 21623 leader_election.cc:290] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [CANDIDATE]: Term 1 election: Requested vote from peers 59025a646f504bcdb79badce6765ca6e (127.20.120.3:46493), 78537f0a126b4fc99a3f75df886f6638 (127.20.120.1:45271)
I20250814 07:57:22.055735 21416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07f9fb6555f4487890d8b6f2b409b59f" candidate_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59025a646f504bcdb79badce6765ca6e"
I20250814 07:57:22.056111 21416 raft_consensus.cc:3058] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:22.056249 21150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07f9fb6555f4487890d8b6f2b409b59f" candidate_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78537f0a126b4fc99a3f75df886f6638"
I20250814 07:57:22.056643 21150 raft_consensus.cc:3058] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:22.060508 21416 raft_consensus.cc:2466] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae448f18b4b49bbb52063c2315a9fb2 in term 1.
I20250814 07:57:22.060885 21150 raft_consensus.cc:2466] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1ae448f18b4b49bbb52063c2315a9fb2 in term 1.
I20250814 07:57:22.061558 21216 leader_election.cc:304] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ae448f18b4b49bbb52063c2315a9fb2, 59025a646f504bcdb79badce6765ca6e; no voters:
I20250814 07:57:22.062173 21623 raft_consensus.cc:2802] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:57:22.063697 21623 raft_consensus.cc:695] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [term 1 LEADER]: Becoming Leader. State: Replica: 1ae448f18b4b49bbb52063c2315a9fb2, State: Running, Role: LEADER
I20250814 07:57:22.064493 21623 consensus_queue.cc:237] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:22.073055 21006 catalog_manager.cc:5582] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2). New cstate: current_term: 1 leader_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } health_report { overall_health: UNKNOWN } } }
I20250814 07:57:22.090682 20960 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250814 07:57:22.094012 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 78537f0a126b4fc99a3f75df886f6638 to finish bootstrapping
W20250814 07:57:22.103825 21462 tablet.cc:2378] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:57:22.107134 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1ae448f18b4b49bbb52063c2315a9fb2 to finish bootstrapping
I20250814 07:57:22.116292 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 59025a646f504bcdb79badce6765ca6e to finish bootstrapping
I20250814 07:57:22.125317 20960 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250814 07:57:22.128139 20960 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "07f9fb6555f4487890d8b6f2b409b59f"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "59025a646f504bcdb79badce6765ca6e"
rpc_addresses {
host: "127.20.120.3"
port: 46493
}
}
ts_infos {
permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2"
rpc_addresses {
host: "127.20.120.2"
port: 41969
}
}
ts_infos {
permanent_uuid: "78537f0a126b4fc99a3f75df886f6638"
rpc_addresses {
host: "127.20.120.1"
port: 45271
}
}
I20250814 07:57:22.445147 21623 consensus_queue.cc:1035] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 07:57:22.460028 21623 consensus_queue.cc:1035] T 07f9fb6555f4487890d8b6f2b409b59f P 1ae448f18b4b49bbb52063c2315a9fb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 07:57:22.465431 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 21199
W20250814 07:57:22.494320 21351 connection.cc:537] server connection from 127.20.120.2:48781 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250814 07:57:22.495682 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 20974
I20250814 07:57:22.524533 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:44207
--webserver_interface=127.20.120.62
--webserver_port=37455
--builtin_ntp_servers=127.20.120.20:46331
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:44207 with env {}
I20250814 07:57:22.712952 21595 heartbeater.cc:499] Master 127.20.120.62:44207 was elected leader, sending a full tablet report...
W20250814 07:57:22.716271 21595 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:44207 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:44207: connect: Connection refused (error 111)
W20250814 07:57:22.818107 21637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:22.818612 21637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:22.819028 21637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:22.847254 21637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:57:22.847525 21637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:22.847757 21637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:57:22.847986 21637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:57:22.880790 21637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:46331
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:44207
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:44207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=37455
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:22.882006 21637 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:22.883565 21637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:22.894460 21644 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:23.488798 21195 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:44207 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:44207: connect: Connection refused (error 111)
W20250814 07:57:23.492192 21461 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:44207 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:44207: connect: Connection refused (error 111)
I20250814 07:57:23.967005 21655 raft_consensus.cc:491] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae448f18b4b49bbb52063c2315a9fb2)
I20250814 07:57:23.969110 21655 raft_consensus.cc:513] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
W20250814 07:57:23.975942 21352 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.2:41969: connect: Connection refused (error 111)
I20250814 07:57:23.976361 21654 raft_consensus.cc:491] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 1ae448f18b4b49bbb52063c2315a9fb2)
I20250814 07:57:23.977162 21654 raft_consensus.cc:513] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:23.995682 21655 leader_election.cc:290] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2:41969), 78537f0a126b4fc99a3f75df886f6638 (127.20.120.1:45271)
W20250814 07:57:24.001113 21352 leader_election.cc:336] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2:41969): Network error: Client connection negotiation failed: client connection to 127.20.120.2:41969: connect: Connection refused (error 111)
I20250814 07:57:24.000473 21654 leader_election.cc:290] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59025a646f504bcdb79badce6765ca6e (127.20.120.3:46493), 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2:41969)
I20250814 07:57:24.015130 21416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07f9fb6555f4487890d8b6f2b409b59f" candidate_uuid: "78537f0a126b4fc99a3f75df886f6638" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59025a646f504bcdb79badce6765ca6e" is_pre_election: true
I20250814 07:57:24.015908 21416 raft_consensus.cc:2466] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78537f0a126b4fc99a3f75df886f6638 in term 1.
I20250814 07:57:24.017884 21083 leader_election.cc:304] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 59025a646f504bcdb79badce6765ca6e, 78537f0a126b4fc99a3f75df886f6638; no voters:
I20250814 07:57:24.019686 21654 raft_consensus.cc:2802] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250814 07:57:24.020184 21654 raft_consensus.cc:491] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 1ae448f18b4b49bbb52063c2315a9fb2)
I20250814 07:57:24.020690 21654 raft_consensus.cc:3058] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 1 FOLLOWER]: Advancing to term 2
W20250814 07:57:24.041976 21086 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.2:41969: connect: Connection refused (error 111)
W20250814 07:57:24.058904 21086 leader_election.cc:336] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2:41969): Network error: Client connection negotiation failed: client connection to 127.20.120.2:41969: connect: Connection refused (error 111)
I20250814 07:57:24.063189 21150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07f9fb6555f4487890d8b6f2b409b59f" candidate_uuid: "59025a646f504bcdb79badce6765ca6e" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "78537f0a126b4fc99a3f75df886f6638" is_pre_election: true
I20250814 07:57:24.117045 21654 raft_consensus.cc:513] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:24.118721 21150 raft_consensus.cc:2391] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 59025a646f504bcdb79badce6765ca6e in current term 2: Already voted for candidate 78537f0a126b4fc99a3f75df886f6638 in this term.
I20250814 07:57:24.124576 21350 leader_election.cc:304] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59025a646f504bcdb79badce6765ca6e; no voters: 1ae448f18b4b49bbb52063c2315a9fb2, 78537f0a126b4fc99a3f75df886f6638
I20250814 07:57:24.126194 21655 raft_consensus.cc:3058] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 1 FOLLOWER]: Advancing to term 2
I20250814 07:57:24.138185 21416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07f9fb6555f4487890d8b6f2b409b59f" candidate_uuid: "78537f0a126b4fc99a3f75df886f6638" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "59025a646f504bcdb79badce6765ca6e"
W20250814 07:57:24.145725 21086 leader_election.cc:336] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2:41969): Network error: Client connection negotiation failed: client connection to 127.20.120.2:41969: connect: Connection refused (error 111)
I20250814 07:57:24.142679 21654 leader_election.cc:290] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [CANDIDATE]: Term 2 election: Requested vote from peers 59025a646f504bcdb79badce6765ca6e (127.20.120.3:46493), 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2:41969)
I20250814 07:57:24.218750 21655 raft_consensus.cc:2747] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250814 07:57:24.324223 21416 raft_consensus.cc:2466] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78537f0a126b4fc99a3f75df886f6638 in term 2.
I20250814 07:57:24.329079 21083 leader_election.cc:304] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 59025a646f504bcdb79badce6765ca6e, 78537f0a126b4fc99a3f75df886f6638; no voters: 1ae448f18b4b49bbb52063c2315a9fb2
I20250814 07:57:24.330741 21654 raft_consensus.cc:2802] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 07:57:24.340256 21654 raft_consensus.cc:695] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 2 LEADER]: Becoming Leader. State: Replica: 78537f0a126b4fc99a3f75df886f6638, State: Running, Role: LEADER
I20250814 07:57:24.343712 21654 consensus_queue.cc:237] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
W20250814 07:57:22.894970 21645 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:24.299774 21643 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 21637
W20250814 07:57:24.715706 21637 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.822s user 0.002s sys 0.000s
W20250814 07:57:24.715991 21646 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1818 milliseconds
W20250814 07:57:24.716131 21637 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.822s user 0.003s sys 0.000s
W20250814 07:57:24.718294 21647 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:24.718231 21637 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:24.721590 21637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:24.724429 21637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:24.725912 21637 hybrid_clock.cc:648] HybridClock initialized: now 1755158244725868 us; error 50 us; skew 500 ppm
I20250814 07:57:24.726753 21637 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:24.733436 21637 webserver.cc:480] Webserver started at http://127.20.120.62:37455/ using document root <none> and password file <none>
I20250814 07:57:24.734699 21637 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:24.735109 21637 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:24.744467 21637 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250814 07:57:24.748792 21669 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:24.749795 21637 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250814 07:57:24.750105 21637 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851"
format_stamp: "Formatted at 2025-08-14 07:57:13 on dist-test-slave-87c1"
I20250814 07:57:24.751942 21637 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:24.799228 21637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:24.800616 21637 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:24.801059 21637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:24.874465 21637 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:44207
I20250814 07:57:24.874507 21720 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:44207 every 8 connection(s)
I20250814 07:57:24.878229 21637 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 07:57:24.879662 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 21637
I20250814 07:57:24.880342 20960 kudu-admin-test.cc:735] Forcing unsafe config change on tserver 78537f0a126b4fc99a3f75df886f6638
I20250814 07:57:24.892899 21721 sys_catalog.cc:263] Verifying existing consensus state
I20250814 07:57:24.897583 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851: Bootstrap starting.
I20250814 07:57:24.940186 21721 log.cc:826] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:24.962996 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:57:24.963769 21721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851: Bootstrap complete.
I20250814 07:57:24.967043 21416 raft_consensus.cc:1273] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 2 FOLLOWER]: Refusing update from remote peer 78537f0a126b4fc99a3f75df886f6638: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250814 07:57:24.969166 21726 consensus_queue.cc:1035] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
W20250814 07:57:24.993901 21086 consensus_peers.cc:489] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 -> Peer 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2:41969): Couldn't send request to peer 1ae448f18b4b49bbb52063c2315a9fb2. Status: Network error: Client connection negotiation failed: client connection to 127.20.120.2:41969: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250814 07:57:25.017592 21721 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } }
I20250814 07:57:25.027381 21721 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bb5c4c525bc442db9a5f0aa0cbf4851, State: Initialized, Role: FOLLOWER
I20250814 07:57:25.028669 21721 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } }
I20250814 07:57:25.029544 21721 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:57:25.029943 21721 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:57:25.030479 21721 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 1 FOLLOWER]: Advancing to term 2
I20250814 07:57:25.041268 21461 heartbeater.cc:344] Connected to a master server at 127.20.120.62:44207
I20250814 07:57:25.042191 21195 heartbeater.cc:344] Connected to a master server at 127.20.120.62:44207
I20250814 07:57:25.136792 21721 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } }
I20250814 07:57:25.137889 21721 leader_election.cc:304] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bb5c4c525bc442db9a5f0aa0cbf4851; no voters:
I20250814 07:57:25.141554 21721 leader_election.cc:290] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250814 07:57:25.141841 21735 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 07:57:25.144323 21735 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [term 2 LEADER]: Becoming Leader. State: Replica: 9bb5c4c525bc442db9a5f0aa0cbf4851, State: Running, Role: LEADER
I20250814 07:57:25.145205 21735 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } }
I20250814 07:57:25.147754 21721 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:57:25.154420 21737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } } }
I20250814 07:57:25.155324 21737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [sys.catalog]: This master's current role is: LEADER
I20250814 07:57:25.155231 21736 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bb5c4c525bc442db9a5f0aa0cbf4851. Latest consensus state: current_term: 2 leader_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bb5c4c525bc442db9a5f0aa0cbf4851" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 44207 } } }
I20250814 07:57:25.155789 21736 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851 [sys.catalog]: This master's current role is: LEADER
I20250814 07:57:25.165844 21742 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:57:25.180186 21742 catalog_manager.cc:671] Loaded metadata for table TestTable [id=873a82748fad4da3a524bc6cc70f63de]
I20250814 07:57:25.187546 21742 tablet_loader.cc:96] loaded metadata for tablet 07f9fb6555f4487890d8b6f2b409b59f (table TestTable [id=873a82748fad4da3a524bc6cc70f63de])
I20250814 07:57:25.189155 21742 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:57:25.193619 21742 catalog_manager.cc:1261] Loaded cluster ID: 77bfc86e448947e5934392fd1e8f2216
I20250814 07:57:25.193898 21742 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:57:25.200974 21742 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:57:25.206175 21742 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 9bb5c4c525bc442db9a5f0aa0cbf4851: Loaded TSK: 0
I20250814 07:57:25.207981 21742 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250814 07:57:25.258177 21723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:25.258749 21723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:25.289582 21723 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250814 07:57:25.768119 21595 heartbeater.cc:344] Connected to a master server at 127.20.120.62:44207
I20250814 07:57:25.768589 21595 heartbeater.cc:499] Master 127.20.120.62:44207 was elected leader, sending a full tablet report...
I20250814 07:57:25.773788 21685 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8748301262c74e6eb010c9e299822208" instance_seqno: 1755158241647716) as {username='slave'} at 127.20.120.4:33627; Asking this server to re-register.
I20250814 07:57:25.774979 21595 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:25.775517 21595 heartbeater.cc:507] Master 127.20.120.62:44207 requested a full tablet report, sending...
I20250814 07:57:25.777653 21685 ts_manager.cc:194] Registered new tserver with Master: 8748301262c74e6eb010c9e299822208 (127.20.120.4:35251)
I20250814 07:57:26.050511 21685 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" instance_seqno: 1755158235632306) as {username='slave'} at 127.20.120.1:36541; Asking this server to re-register.
I20250814 07:57:26.052238 21195 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:26.052932 21195 heartbeater.cc:507] Master 127.20.120.62:44207 requested a full tablet report, sending...
I20250814 07:57:26.053052 21685 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "59025a646f504bcdb79badce6765ca6e" instance_seqno: 1755158239814011) as {username='slave'} at 127.20.120.3:59187; Asking this server to re-register.
I20250814 07:57:26.054574 21461 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:26.055202 21461 heartbeater.cc:507] Master 127.20.120.62:44207 requested a full tablet report, sending...
I20250814 07:57:26.057966 21686 ts_manager.cc:194] Registered new tserver with Master: 59025a646f504bcdb79badce6765ca6e (127.20.120.3:46493)
I20250814 07:57:26.062898 21685 ts_manager.cc:194] Registered new tserver with Master: 78537f0a126b4fc99a3f75df886f6638 (127.20.120.1:45271)
I20250814 07:57:26.064200 21686 catalog_manager.cc:5582] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e reported cstate change: term changed from 1 to 2, leader changed from 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2) to 78537f0a126b4fc99a3f75df886f6638 (127.20.120.1). New cstate: current_term: 2 leader_uuid: "78537f0a126b4fc99a3f75df886f6638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } }
W20250814 07:57:26.733251 21758 debug-util.cc:398] Leaking SignalData structure 0x7b08000347a0 after lost signal to thread 21723
W20250814 07:57:26.733755 21758 kernel_stack_watchdog.cc:198] Thread 21723 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 07:57:26.920814 21723 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.590s user 0.538s sys 0.990s
W20250814 07:57:27.037979 21723 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.707s user 0.549s sys 0.993s
I20250814 07:57:27.083050 21150 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "78537f0a126b4fc99a3f75df886f6638"
tablet_id: "07f9fb6555f4487890d8b6f2b409b59f"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "59025a646f504bcdb79badce6765ca6e"
}
peers {
permanent_uuid: "78537f0a126b4fc99a3f75df886f6638"
}
}
from {username='slave'} at 127.0.0.1:36288
W20250814 07:57:27.084061 21150 raft_consensus.cc:2216] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 2 LEADER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } unsafe_config_change: true
I20250814 07:57:27.084859 21150 raft_consensus.cc:3053] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 2 LEADER]: Stepping down as leader of term 2
I20250814 07:57:27.085129 21150 raft_consensus.cc:738] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 78537f0a126b4fc99a3f75df886f6638, State: Running, Role: LEADER
I20250814 07:57:27.085772 21150 consensus_queue.cc:260] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:27.086673 21150 raft_consensus.cc:3058] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 2 FOLLOWER]: Advancing to term 3
I20250814 07:57:28.564534 21785 raft_consensus.cc:491] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 78537f0a126b4fc99a3f75df886f6638)
I20250814 07:57:28.565135 21785 raft_consensus.cc:513] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } }
I20250814 07:57:28.566707 21785 leader_election.cc:290] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2:41969), 78537f0a126b4fc99a3f75df886f6638 (127.20.120.1:45271)
I20250814 07:57:28.567782 21150 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07f9fb6555f4487890d8b6f2b409b59f" candidate_uuid: "59025a646f504bcdb79badce6765ca6e" candidate_term: 3 candidate_status { last_received { term: 2 index: 2 } } ignore_live_leader: false dest_uuid: "78537f0a126b4fc99a3f75df886f6638" is_pre_election: true
W20250814 07:57:28.572412 21352 leader_election.cc:336] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2:41969): Network error: Client connection negotiation failed: client connection to 127.20.120.2:41969: connect: Connection refused (error 111)
I20250814 07:57:28.572727 21352 leader_election.cc:304] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 59025a646f504bcdb79badce6765ca6e; no voters: 1ae448f18b4b49bbb52063c2315a9fb2, 78537f0a126b4fc99a3f75df886f6638
I20250814 07:57:28.573266 21785 raft_consensus.cc:2747] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250814 07:57:28.597257 21788 raft_consensus.cc:491] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250814 07:57:28.597575 21788 raft_consensus.cc:513] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } unsafe_config_change: true
I20250814 07:57:28.598529 21788 leader_election.cc:290] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 59025a646f504bcdb79badce6765ca6e (127.20.120.3:46493)
I20250814 07:57:28.599575 21416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07f9fb6555f4487890d8b6f2b409b59f" candidate_uuid: "78537f0a126b4fc99a3f75df886f6638" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59025a646f504bcdb79badce6765ca6e" is_pre_election: true
I20250814 07:57:28.600066 21416 raft_consensus.cc:2466] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78537f0a126b4fc99a3f75df886f6638 in term 2.
I20250814 07:57:28.600911 21083 leader_election.cc:304] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 59025a646f504bcdb79badce6765ca6e, 78537f0a126b4fc99a3f75df886f6638; no voters:
I20250814 07:57:28.601440 21788 raft_consensus.cc:2802] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250814 07:57:28.601676 21788 raft_consensus.cc:491] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250814 07:57:28.601882 21788 raft_consensus.cc:3058] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 3 FOLLOWER]: Advancing to term 4
I20250814 07:57:28.605775 21788 raft_consensus.cc:513] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } unsafe_config_change: true
I20250814 07:57:28.606621 21788 leader_election.cc:290] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [CANDIDATE]: Term 4 election: Requested vote from peers 59025a646f504bcdb79badce6765ca6e (127.20.120.3:46493)
I20250814 07:57:28.607405 21416 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "07f9fb6555f4487890d8b6f2b409b59f" candidate_uuid: "78537f0a126b4fc99a3f75df886f6638" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "59025a646f504bcdb79badce6765ca6e"
I20250814 07:57:28.607741 21416 raft_consensus.cc:3058] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 2 FOLLOWER]: Advancing to term 4
I20250814 07:57:28.612071 21416 raft_consensus.cc:2466] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78537f0a126b4fc99a3f75df886f6638 in term 4.
I20250814 07:57:28.612752 21083 leader_election.cc:304] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 59025a646f504bcdb79badce6765ca6e, 78537f0a126b4fc99a3f75df886f6638; no voters:
I20250814 07:57:28.613270 21788 raft_consensus.cc:2802] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 4 FOLLOWER]: Leader election won for term 4
I20250814 07:57:28.613940 21788 raft_consensus.cc:695] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 4 LEADER]: Becoming Leader. State: Replica: 78537f0a126b4fc99a3f75df886f6638, State: Running, Role: LEADER
I20250814 07:57:28.614437 21788 consensus_queue.cc:237] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } unsafe_config_change: true
I20250814 07:57:28.619449 21685 catalog_manager.cc:5582] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 reported cstate change: term changed from 2 to 4, now has a pending config: VOTER 59025a646f504bcdb79badce6765ca6e (127.20.120.3), VOTER 78537f0a126b4fc99a3f75df886f6638 (127.20.120.1). New cstate: current_term: 4 leader_uuid: "78537f0a126b4fc99a3f75df886f6638" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ae448f18b4b49bbb52063c2315a9fb2" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 41969 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } health_report { overall_health: HEALTHY } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } unsafe_config_change: true }
I20250814 07:57:29.160058 21416 raft_consensus.cc:1273] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 4 FOLLOWER]: Refusing update from remote peer 78537f0a126b4fc99a3f75df886f6638: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250814 07:57:29.161222 21795 consensus_queue.cc:1035] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250814 07:57:29.168848 21798 raft_consensus.cc:2953] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } unsafe_config_change: true }
I20250814 07:57:29.169725 21416 raft_consensus.cc:2953] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } unsafe_config_change: true }
I20250814 07:57:29.179322 21685 catalog_manager.cc:5582] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e reported cstate change: config changed from index -1 to 3, VOTER 1ae448f18b4b49bbb52063c2315a9fb2 (127.20.120.2) evicted, no longer has a pending config: VOTER 59025a646f504bcdb79badce6765ca6e (127.20.120.3), VOTER 78537f0a126b4fc99a3f75df886f6638 (127.20.120.1). New cstate: current_term: 4 leader_uuid: "78537f0a126b4fc99a3f75df886f6638" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } unsafe_config_change: true }
W20250814 07:57:29.186965 21685 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 07f9fb6555f4487890d8b6f2b409b59f on TS 1ae448f18b4b49bbb52063c2315a9fb2: Not found: failed to reset TS proxy: Could not find TS for UUID 1ae448f18b4b49bbb52063c2315a9fb2
I20250814 07:57:29.200553 21150 consensus_queue.cc:237] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: NON_VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: true } } unsafe_config_change: true
I20250814 07:57:29.204602 21416 raft_consensus.cc:1273] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 4 FOLLOWER]: Refusing update from remote peer 78537f0a126b4fc99a3f75df886f6638: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250814 07:57:29.206005 21795 consensus_queue.cc:1035] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250814 07:57:29.210426 21798 raft_consensus.cc:2953] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 8748301262c74e6eb010c9e299822208 (127.20.120.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: NON_VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: true } } unsafe_config_change: true }
I20250814 07:57:29.211735 21416 raft_consensus.cc:2953] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER 8748301262c74e6eb010c9e299822208 (127.20.120.4) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: NON_VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: true } } unsafe_config_change: true }
W20250814 07:57:29.214490 21085 consensus_peers.cc:489] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 -> Peer 8748301262c74e6eb010c9e299822208 (127.20.120.4:35251): Couldn't send request to peer 8748301262c74e6eb010c9e299822208. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 07f9fb6555f4487890d8b6f2b409b59f. This is attempt 1: this message will repeat every 5th retry.
I20250814 07:57:29.218171 21671 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 07f9fb6555f4487890d8b6f2b409b59f with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250814 07:57:29.221312 21685 catalog_manager.cc:5582] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 reported cstate change: config changed from index 3 to 5, NON_VOTER 8748301262c74e6eb010c9e299822208 (127.20.120.4) added. New cstate: current_term: 4 leader_uuid: "78537f0a126b4fc99a3f75df886f6638" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: NON_VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250814 07:57:29.239560 21671 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 07f9fb6555f4487890d8b6f2b409b59f on TS 1ae448f18b4b49bbb52063c2315a9fb2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 1ae448f18b4b49bbb52063c2315a9fb2
I20250814 07:57:29.628979 21808 ts_tablet_manager.cc:927] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: Initiating tablet copy from peer 78537f0a126b4fc99a3f75df886f6638 (127.20.120.1:45271)
I20250814 07:57:29.631012 21808 tablet_copy_client.cc:323] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: tablet copy: Beginning tablet copy session from remote peer at address 127.20.120.1:45271
I20250814 07:57:29.640587 21170 tablet_copy_service.cc:140] P 78537f0a126b4fc99a3f75df886f6638: Received BeginTabletCopySession request for tablet 07f9fb6555f4487890d8b6f2b409b59f from peer 8748301262c74e6eb010c9e299822208 ({username='slave'} at 127.20.120.4:33063)
I20250814 07:57:29.641031 21170 tablet_copy_service.cc:161] P 78537f0a126b4fc99a3f75df886f6638: Beginning new tablet copy session on tablet 07f9fb6555f4487890d8b6f2b409b59f from peer 8748301262c74e6eb010c9e299822208 at {username='slave'} at 127.20.120.4:33063: session id = 8748301262c74e6eb010c9e299822208-07f9fb6555f4487890d8b6f2b409b59f
I20250814 07:57:29.647253 21170 tablet_copy_source_session.cc:215] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638: Tablet Copy: opened 0 blocks and 1 log segments
I20250814 07:57:29.651898 21808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 07f9fb6555f4487890d8b6f2b409b59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:29.668586 21808 tablet_copy_client.cc:806] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: tablet copy: Starting download of 0 data blocks...
I20250814 07:57:29.669126 21808 tablet_copy_client.cc:670] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: tablet copy: Starting download of 1 WAL segments...
I20250814 07:57:29.672061 21808 tablet_copy_client.cc:538] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250814 07:57:29.679409 21808 tablet_bootstrap.cc:492] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: Bootstrap starting.
I20250814 07:57:29.693457 21808 log.cc:826] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:29.707094 21808 tablet_bootstrap.cc:492] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:57:29.707892 21808 tablet_bootstrap.cc:492] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: Bootstrap complete.
I20250814 07:57:29.708520 21808 ts_tablet_manager.cc:1397] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: Time spent bootstrapping tablet: real 0.029s user 0.022s sys 0.008s
I20250814 07:57:29.730872 21808 raft_consensus.cc:357] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: NON_VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: true } } unsafe_config_change: true
I20250814 07:57:29.731547 21808 raft_consensus.cc:738] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: 8748301262c74e6eb010c9e299822208, State: Initialized, Role: LEARNER
I20250814 07:57:29.732110 21808 consensus_queue.cc:260] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: NON_VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: true } } unsafe_config_change: true
I20250814 07:57:29.735112 21808 ts_tablet_manager.cc:1428] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208: Time spent starting tablet: real 0.026s user 0.018s sys 0.007s
I20250814 07:57:29.736779 21170 tablet_copy_service.cc:342] P 78537f0a126b4fc99a3f75df886f6638: Request end of tablet copy session 8748301262c74e6eb010c9e299822208-07f9fb6555f4487890d8b6f2b409b59f received from {username='slave'} at 127.20.120.4:33063
I20250814 07:57:29.737262 21170 tablet_copy_service.cc:434] P 78537f0a126b4fc99a3f75df886f6638: ending tablet copy session 8748301262c74e6eb010c9e299822208-07f9fb6555f4487890d8b6f2b409b59f on tablet 07f9fb6555f4487890d8b6f2b409b59f with peer 8748301262c74e6eb010c9e299822208
I20250814 07:57:30.116403 21550 raft_consensus.cc:1215] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250814 07:57:30.406085 21671 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 07f9fb6555f4487890d8b6f2b409b59f on TS 1ae448f18b4b49bbb52063c2315a9fb2 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 1ae448f18b4b49bbb52063c2315a9fb2
I20250814 07:57:30.673604 21814 raft_consensus.cc:1062] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638: attempting to promote NON_VOTER 8748301262c74e6eb010c9e299822208 to VOTER
I20250814 07:57:30.675482 21814 consensus_queue.cc:237] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: false } } unsafe_config_change: true
I20250814 07:57:30.680868 21416 raft_consensus.cc:1273] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 4 FOLLOWER]: Refusing update from remote peer 78537f0a126b4fc99a3f75df886f6638: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250814 07:57:30.681233 21550 raft_consensus.cc:1273] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208 [term 4 LEARNER]: Refusing update from remote peer 78537f0a126b4fc99a3f75df886f6638: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250814 07:57:30.682154 21814 consensus_queue.cc:1035] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250814 07:57:30.682883 21817 consensus_queue.cc:1035] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250814 07:57:30.699808 21817 raft_consensus.cc:2953] T 07f9fb6555f4487890d8b6f2b409b59f P 78537f0a126b4fc99a3f75df886f6638 [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 8748301262c74e6eb010c9e299822208 (127.20.120.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: false } } unsafe_config_change: true }
I20250814 07:57:30.701480 21415 raft_consensus.cc:2953] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 8748301262c74e6eb010c9e299822208 (127.20.120.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: false } } unsafe_config_change: true }
I20250814 07:57:30.710783 21550 raft_consensus.cc:2953] T 07f9fb6555f4487890d8b6f2b409b59f P 8748301262c74e6eb010c9e299822208 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, 8748301262c74e6eb010c9e299822208 (127.20.120.4) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: false } } unsafe_config_change: true }
I20250814 07:57:30.718015 21685 catalog_manager.cc:5582] T 07f9fb6555f4487890d8b6f2b409b59f P 59025a646f504bcdb79badce6765ca6e reported cstate change: config changed from index 5 to 6, 8748301262c74e6eb010c9e299822208 (127.20.120.4) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "78537f0a126b4fc99a3f75df886f6638" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "59025a646f504bcdb79badce6765ca6e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46493 } } peers { permanent_uuid: "78537f0a126b4fc99a3f75df886f6638" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45271 } } peers { permanent_uuid: "8748301262c74e6eb010c9e299822208" member_type: VOTER last_known_addr { host: "127.20.120.4" port: 35251 } attrs { promote: false } } unsafe_config_change: true }
I20250814 07:57:30.749758 20960 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250814 07:57:30.762657 20960 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "07f9fb6555f4487890d8b6f2b409b59f"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 1
role: LEADER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "59025a646f504bcdb79badce6765ca6e"
rpc_addresses {
host: "127.20.120.3"
port: 46493
}
}
ts_infos {
permanent_uuid: "78537f0a126b4fc99a3f75df886f6638"
rpc_addresses {
host: "127.20.120.1"
port: 45271
}
}
ts_infos {
permanent_uuid: "8748301262c74e6eb010c9e299822208"
rpc_addresses {
host: "127.20.120.4"
port: 35251
}
}
I20250814 07:57:30.765360 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 21066
I20250814 07:57:30.791664 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 21332
I20250814 07:57:30.815327 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 21465
I20250814 07:57:30.838114 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 21637
2025-08-14T07:57:30Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (20605 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250814 07:57:30.899748 20960 test_util.cc:276] Using random seed: -1609447160
I20250814 07:57:30.907680 20960 ts_itest-base.cc:115] Starting cluster with:
I20250814 07:57:30.907877 20960 ts_itest-base.cc:116] --------------
I20250814 07:57:30.908067 20960 ts_itest-base.cc:117] 3 tablet servers
I20250814 07:57:30.908257 20960 ts_itest-base.cc:118] 3 replicas per TS
I20250814 07:57:30.908407 20960 ts_itest-base.cc:119] --------------
2025-08-14T07:57:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T07:57:30Z Disabled control of system clock
I20250814 07:57:30.940992 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:45079
--webserver_interface=127.20.120.62
--webserver_port=0
--builtin_ntp_servers=127.20.120.20:37439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:45079
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250814 07:57:31.222976 21837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:31.223488 21837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:31.223892 21837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:31.253228 21837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:57:31.253502 21837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:31.253710 21837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:57:31.253911 21837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:57:31.287189 21837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:37439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:45079
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:45079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:31.288370 21837 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:31.289903 21837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:31.300315 21843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:31.302021 21844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:32.475025 21846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:32.477263 21845 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1171 milliseconds
I20250814 07:57:32.477384 21837 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:32.478590 21837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:32.481359 21837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:32.482674 21837 hybrid_clock.cc:648] HybridClock initialized: now 1755158252482635 us; error 44 us; skew 500 ppm
I20250814 07:57:32.483383 21837 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:32.489035 21837 webserver.cc:480] Webserver started at http://127.20.120.62:42119/ using document root <none> and password file <none>
I20250814 07:57:32.489882 21837 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:32.490077 21837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:32.490494 21837 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:32.494671 21837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "0e5f45ea77a348688b11fa20c3bf7b20"
format_stamp: "Formatted at 2025-08-14 07:57:32 on dist-test-slave-87c1"
I20250814 07:57:32.495704 21837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "0e5f45ea77a348688b11fa20c3bf7b20"
format_stamp: "Formatted at 2025-08-14 07:57:32 on dist-test-slave-87c1"
I20250814 07:57:32.502310 21837 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250814 07:57:32.507318 21853 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:32.508237 21837 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250814 07:57:32.508520 21837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "0e5f45ea77a348688b11fa20c3bf7b20"
format_stamp: "Formatted at 2025-08-14 07:57:32 on dist-test-slave-87c1"
I20250814 07:57:32.508822 21837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:32.564340 21837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:32.565723 21837 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:32.566112 21837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:32.633540 21837 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:45079
I20250814 07:57:32.633600 21904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:45079 every 8 connection(s)
I20250814 07:57:32.636210 21837 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 07:57:32.636910 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 21837
I20250814 07:57:32.637308 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 07:57:32.643442 21905 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:32.666067 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20: Bootstrap starting.
I20250814 07:57:32.670909 21905 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:32.672365 21905 log.cc:826] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:32.676110 21905 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20: No bootstrap required, opened a new log
I20250814 07:57:32.691342 21905 raft_consensus.cc:357] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5f45ea77a348688b11fa20c3bf7b20" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 45079 } }
I20250814 07:57:32.691944 21905 raft_consensus.cc:383] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:32.692160 21905 raft_consensus.cc:738] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5f45ea77a348688b11fa20c3bf7b20, State: Initialized, Role: FOLLOWER
I20250814 07:57:32.692740 21905 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5f45ea77a348688b11fa20c3bf7b20" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 45079 } }
I20250814 07:57:32.693280 21905 raft_consensus.cc:397] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:57:32.693527 21905 raft_consensus.cc:491] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:57:32.693822 21905 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:32.697490 21905 raft_consensus.cc:513] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5f45ea77a348688b11fa20c3bf7b20" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 45079 } }
I20250814 07:57:32.698163 21905 leader_election.cc:304] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e5f45ea77a348688b11fa20c3bf7b20; no voters:
I20250814 07:57:32.699641 21905 leader_election.cc:290] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:57:32.700383 21910 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:57:32.702512 21910 raft_consensus.cc:695] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [term 1 LEADER]: Becoming Leader. State: Replica: 0e5f45ea77a348688b11fa20c3bf7b20, State: Running, Role: LEADER
I20250814 07:57:32.703207 21910 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5f45ea77a348688b11fa20c3bf7b20" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 45079 } }
I20250814 07:57:32.704164 21905 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:57:32.713203 21912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e5f45ea77a348688b11fa20c3bf7b20. Latest consensus state: current_term: 1 leader_uuid: "0e5f45ea77a348688b11fa20c3bf7b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5f45ea77a348688b11fa20c3bf7b20" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 45079 } } }
I20250814 07:57:32.713362 21911 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e5f45ea77a348688b11fa20c3bf7b20" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5f45ea77a348688b11fa20c3bf7b20" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 45079 } } }
I20250814 07:57:32.713902 21912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [sys.catalog]: This master's current role is: LEADER
I20250814 07:57:32.713917 21911 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20 [sys.catalog]: This master's current role is: LEADER
I20250814 07:57:32.716977 21919 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:57:32.728099 21919 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:57:32.743670 21919 catalog_manager.cc:1349] Generated new cluster ID: dc51e487f08c4322bb89df3774384891
I20250814 07:57:32.743955 21919 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:57:32.770849 21919 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 07:57:32.772198 21919 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:57:32.784057 21919 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 0e5f45ea77a348688b11fa20c3bf7b20: Generated new TSK 0
I20250814 07:57:32.784840 21919 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 07:57:32.810352 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:0
--local_ip_for_outbound_sockets=127.20.120.1
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:45079
--builtin_ntp_servers=127.20.120.20:37439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250814 07:57:33.117259 21929 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250814 07:57:33.117866 21929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:33.118162 21929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:33.118650 21929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:33.147430 21929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:33.148206 21929 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 07:57:33.179960 21929 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:37439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:45079
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:33.181120 21929 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:33.182502 21929 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:33.194175 21935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:34.595405 21934 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 21929
W20250814 07:57:34.865303 21929 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.670s user 0.572s sys 1.054s
W20250814 07:57:33.195719 21936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:34.866322 21929 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.672s user 0.573s sys 1.054s
W20250814 07:57:34.867136 21937 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1670 milliseconds
W20250814 07:57:34.867347 21938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:34.867794 21929 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:34.870775 21929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:34.872848 21929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:34.874210 21929 hybrid_clock.cc:648] HybridClock initialized: now 1755158254874165 us; error 49 us; skew 500 ppm
I20250814 07:57:34.874971 21929 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:34.880923 21929 webserver.cc:480] Webserver started at http://127.20.120.1:37645/ using document root <none> and password file <none>
I20250814 07:57:34.881857 21929 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:34.882077 21929 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:34.882511 21929 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:34.886756 21929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "9772a83eea10458f835e5cf72ebd0c8a"
format_stamp: "Formatted at 2025-08-14 07:57:34 on dist-test-slave-87c1"
I20250814 07:57:34.887814 21929 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "9772a83eea10458f835e5cf72ebd0c8a"
format_stamp: "Formatted at 2025-08-14 07:57:34 on dist-test-slave-87c1"
I20250814 07:57:34.894877 21929 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250814 07:57:34.900894 21945 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:34.902357 21929 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250814 07:57:34.902830 21929 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "9772a83eea10458f835e5cf72ebd0c8a"
format_stamp: "Formatted at 2025-08-14 07:57:34 on dist-test-slave-87c1"
I20250814 07:57:34.903378 21929 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:34.953289 21929 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:34.954648 21929 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:34.955036 21929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:34.957451 21929 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:57:34.961261 21929 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:57:34.961443 21929 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:34.961665 21929 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:57:34.961817 21929 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:35.120000 21929 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:36635
I20250814 07:57:35.120169 22057 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:36635 every 8 connection(s)
I20250814 07:57:35.122545 21929 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 07:57:35.128643 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 21929
I20250814 07:57:35.129108 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 07:57:35.135478 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.2:0
--local_ip_for_outbound_sockets=127.20.120.2
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:45079
--builtin_ntp_servers=127.20.120.20:37439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250814 07:57:35.150209 22058 heartbeater.cc:344] Connected to a master server at 127.20.120.62:45079
I20250814 07:57:35.150720 22058 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:35.152010 22058 heartbeater.cc:507] Master 127.20.120.62:45079 requested a full tablet report, sending...
I20250814 07:57:35.155107 21870 ts_manager.cc:194] Registered new tserver with Master: 9772a83eea10458f835e5cf72ebd0c8a (127.20.120.1:36635)
I20250814 07:57:35.157701 21870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:48947
W20250814 07:57:35.429090 22062 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250814 07:57:35.429668 22062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:35.429934 22062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:35.430414 22062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:35.460083 22062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:35.460878 22062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.2
I20250814 07:57:35.492575 22062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:37439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:45079
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.2
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:35.493794 22062 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:35.495232 22062 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:35.506249 22068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:36.161693 22058 heartbeater.cc:499] Master 127.20.120.62:45079 was elected leader, sending a full tablet report...
W20250814 07:57:35.506731 22069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:36.660742 22071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:36.664300 22070 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1153 milliseconds
I20250814 07:57:36.664426 22062 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:36.665540 22062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:36.668148 22062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:36.669552 22062 hybrid_clock.cc:648] HybridClock initialized: now 1755158256669514 us; error 41 us; skew 500 ppm
I20250814 07:57:36.670306 22062 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:36.677099 22062 webserver.cc:480] Webserver started at http://127.20.120.2:33661/ using document root <none> and password file <none>
I20250814 07:57:36.678013 22062 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:36.678231 22062 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:36.678692 22062 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:36.682932 22062 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "95fd1736feca440c90bdd3e9dcd04340"
format_stamp: "Formatted at 2025-08-14 07:57:36 on dist-test-slave-87c1"
I20250814 07:57:36.683987 22062 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "95fd1736feca440c90bdd3e9dcd04340"
format_stamp: "Formatted at 2025-08-14 07:57:36 on dist-test-slave-87c1"
I20250814 07:57:36.690418 22062 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250814 07:57:36.695709 22078 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:36.696609 22062 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250814 07:57:36.696905 22062 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "95fd1736feca440c90bdd3e9dcd04340"
format_stamp: "Formatted at 2025-08-14 07:57:36 on dist-test-slave-87c1"
I20250814 07:57:36.697235 22062 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:36.747853 22062 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:36.749197 22062 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:36.749600 22062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:36.751905 22062 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:57:36.755704 22062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:57:36.755895 22062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:36.756153 22062 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:57:36.756320 22062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:36.883582 22062 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.2:34701
I20250814 07:57:36.883687 22190 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.2:34701 every 8 connection(s)
I20250814 07:57:36.886045 22062 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 07:57:36.892309 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 22062
I20250814 07:57:36.892704 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 07:57:36.898748 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.3:0
--local_ip_for_outbound_sockets=127.20.120.3
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:45079
--builtin_ntp_servers=127.20.120.20:37439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250814 07:57:36.910975 22191 heartbeater.cc:344] Connected to a master server at 127.20.120.62:45079
I20250814 07:57:36.911361 22191 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:36.912308 22191 heartbeater.cc:507] Master 127.20.120.62:45079 requested a full tablet report, sending...
I20250814 07:57:36.914515 21870 ts_manager.cc:194] Registered new tserver with Master: 95fd1736feca440c90bdd3e9dcd04340 (127.20.120.2:34701)
I20250814 07:57:36.916275 21870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.2:53903
W20250814 07:57:37.198143 22195 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250814 07:57:37.198670 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:37.198880 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:37.199261 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:37.228885 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:37.229709 22195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.3
I20250814 07:57:37.263373 22195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:37439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:45079
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.3
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:37.264519 22195 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:37.266031 22195 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:37.279754 22201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:37.919436 22191 heartbeater.cc:499] Master 127.20.120.62:45079 was elected leader, sending a full tablet report...
W20250814 07:57:37.281390 22202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:37.283488 22195 server_base.cc:1047] running on GCE node
W20250814 07:57:37.283231 22204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:38.416899 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:38.419049 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:38.420378 22195 hybrid_clock.cc:648] HybridClock initialized: now 1755158258420335 us; error 52 us; skew 500 ppm
I20250814 07:57:38.421200 22195 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:38.427572 22195 webserver.cc:480] Webserver started at http://127.20.120.3:34901/ using document root <none> and password file <none>
I20250814 07:57:38.428421 22195 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:38.428633 22195 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:38.429154 22195 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:38.433629 22195 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "9ec20e75928d47a99878f024adc93ed6"
format_stamp: "Formatted at 2025-08-14 07:57:38 on dist-test-slave-87c1"
I20250814 07:57:38.434674 22195 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "9ec20e75928d47a99878f024adc93ed6"
format_stamp: "Formatted at 2025-08-14 07:57:38 on dist-test-slave-87c1"
I20250814 07:57:38.441519 22195 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20250814 07:57:38.447021 22211 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:38.447971 22195 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250814 07:57:38.448261 22195 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "9ec20e75928d47a99878f024adc93ed6"
format_stamp: "Formatted at 2025-08-14 07:57:38 on dist-test-slave-87c1"
I20250814 07:57:38.448570 22195 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:38.503019 22195 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:38.504294 22195 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:38.504683 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:38.506979 22195 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:57:38.510757 22195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:57:38.510985 22195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:38.511227 22195 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:57:38.511382 22195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:38.639909 22195 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.3:46151
I20250814 07:57:38.640121 22323 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.3:46151 every 8 connection(s)
I20250814 07:57:38.642639 22195 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 07:57:38.645928 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 22195
I20250814 07:57:38.646332 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 07:57:38.680239 22324 heartbeater.cc:344] Connected to a master server at 127.20.120.62:45079
I20250814 07:57:38.680737 22324 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:38.681739 22324 heartbeater.cc:507] Master 127.20.120.62:45079 requested a full tablet report, sending...
I20250814 07:57:38.683655 21870 ts_manager.cc:194] Registered new tserver with Master: 9ec20e75928d47a99878f024adc93ed6 (127.20.120.3:46151)
I20250814 07:57:38.684903 21870 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.3:50743
I20250814 07:57:38.692483 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:57:38.721210 21870 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34910:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250814 07:57:38.739347 21870 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 07:57:38.784468 22126 tablet_service.cc:1468] Processing CreateTablet for tablet 4aab946a844c42739e52573e8320599a (DEFAULT_TABLE table=TestTable [id=ec27ef840f3b4744b1843fa6ca0286b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:57:38.784358 21993 tablet_service.cc:1468] Processing CreateTablet for tablet 4aab946a844c42739e52573e8320599a (DEFAULT_TABLE table=TestTable [id=ec27ef840f3b4744b1843fa6ca0286b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:57:38.786208 22126 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aab946a844c42739e52573e8320599a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:38.786206 21993 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aab946a844c42739e52573e8320599a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:38.789726 22259 tablet_service.cc:1468] Processing CreateTablet for tablet 4aab946a844c42739e52573e8320599a (DEFAULT_TABLE table=TestTable [id=ec27ef840f3b4744b1843fa6ca0286b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:57:38.790951 22259 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4aab946a844c42739e52573e8320599a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:38.810055 22343 tablet_bootstrap.cc:492] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6: Bootstrap starting.
I20250814 07:57:38.815893 22343 tablet_bootstrap.cc:654] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:38.816613 22345 tablet_bootstrap.cc:492] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a: Bootstrap starting.
I20250814 07:57:38.817982 22343 log.cc:826] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:38.818202 22344 tablet_bootstrap.cc:492] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340: Bootstrap starting.
I20250814 07:57:38.823438 22345 tablet_bootstrap.cc:654] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:38.823508 22343 tablet_bootstrap.cc:492] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6: No bootstrap required, opened a new log
I20250814 07:57:38.824039 22343 ts_tablet_manager.cc:1397] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.005s
I20250814 07:57:38.824959 22344 tablet_bootstrap.cc:654] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:38.825692 22345 log.cc:826] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:38.827278 22344 log.cc:826] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:38.830673 22345 tablet_bootstrap.cc:492] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a: No bootstrap required, opened a new log
I20250814 07:57:38.831022 22345 ts_tablet_manager.cc:1397] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20250814 07:57:38.832425 22344 tablet_bootstrap.cc:492] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340: No bootstrap required, opened a new log
I20250814 07:57:38.832875 22344 ts_tablet_manager.cc:1397] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20250814 07:57:38.848471 22345 raft_consensus.cc:357] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:38.849251 22345 raft_consensus.cc:738] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9772a83eea10458f835e5cf72ebd0c8a, State: Initialized, Role: FOLLOWER
I20250814 07:57:38.849233 22343 raft_consensus.cc:357] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:38.849771 22345 consensus_queue.cc:260] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:38.850052 22343 raft_consensus.cc:738] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec20e75928d47a99878f024adc93ed6, State: Initialized, Role: FOLLOWER
I20250814 07:57:38.850736 22343 consensus_queue.cc:260] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:38.853869 22324 heartbeater.cc:499] Master 127.20.120.62:45079 was elected leader, sending a full tablet report...
I20250814 07:57:38.855257 22343 ts_tablet_manager.cc:1428] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6: Time spent starting tablet: real 0.031s user 0.026s sys 0.003s
I20250814 07:57:38.855890 22345 ts_tablet_manager.cc:1428] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a: Time spent starting tablet: real 0.025s user 0.017s sys 0.004s
I20250814 07:57:38.858903 22344 raft_consensus.cc:357] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:38.859536 22344 raft_consensus.cc:738] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95fd1736feca440c90bdd3e9dcd04340, State: Initialized, Role: FOLLOWER
I20250814 07:57:38.860029 22344 consensus_queue.cc:260] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:38.863195 22344 ts_tablet_manager.cc:1428] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340: Time spent starting tablet: real 0.030s user 0.021s sys 0.007s
W20250814 07:57:38.883023 22059 tablet.cc:2378] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:57:38.885797 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:57:38.889056 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9772a83eea10458f835e5cf72ebd0c8a to finish bootstrapping
W20250814 07:57:38.892132 22192 tablet.cc:2378] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 07:57:38.898164 22325 tablet.cc:2378] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:57:38.905716 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 95fd1736feca440c90bdd3e9dcd04340 to finish bootstrapping
I20250814 07:57:38.919008 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9ec20e75928d47a99878f024adc93ed6 to finish bootstrapping
I20250814 07:57:38.958596 22013 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4aab946a844c42739e52573e8320599a"
dest_uuid: "9772a83eea10458f835e5cf72ebd0c8a"
from {username='slave'} at 127.0.0.1:54258
I20250814 07:57:38.959143 22013 raft_consensus.cc:491] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250814 07:57:38.959441 22013 raft_consensus.cc:3058] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:38.963757 22013 raft_consensus.cc:513] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:38.965884 22013 leader_election.cc:290] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [CANDIDATE]: Term 1 election: Requested vote from peers 95fd1736feca440c90bdd3e9dcd04340 (127.20.120.2:34701), 9ec20e75928d47a99878f024adc93ed6 (127.20.120.3:46151)
I20250814 07:57:38.974733 20960 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250814 07:57:38.978909 22279 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4aab946a844c42739e52573e8320599a" candidate_uuid: "9772a83eea10458f835e5cf72ebd0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9ec20e75928d47a99878f024adc93ed6"
I20250814 07:57:38.978916 22146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4aab946a844c42739e52573e8320599a" candidate_uuid: "9772a83eea10458f835e5cf72ebd0c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "95fd1736feca440c90bdd3e9dcd04340"
I20250814 07:57:38.979635 22279 raft_consensus.cc:3058] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:38.979631 22146 raft_consensus.cc:3058] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:38.986217 22146 raft_consensus.cc:2466] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9772a83eea10458f835e5cf72ebd0c8a in term 1.
I20250814 07:57:38.986231 22279 raft_consensus.cc:2466] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9772a83eea10458f835e5cf72ebd0c8a in term 1.
I20250814 07:57:38.987248 21948 leader_election.cc:304] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9772a83eea10458f835e5cf72ebd0c8a, 9ec20e75928d47a99878f024adc93ed6; no voters:
I20250814 07:57:38.987984 22349 raft_consensus.cc:2802] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:57:38.989625 22349 raft_consensus.cc:695] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 1 LEADER]: Becoming Leader. State: Replica: 9772a83eea10458f835e5cf72ebd0c8a, State: Running, Role: LEADER
I20250814 07:57:38.990386 22349 consensus_queue.cc:237] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:39.000741 21868 catalog_manager.cc:5582] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a reported cstate change: term changed from 0 to 1, leader changed from <none> to 9772a83eea10458f835e5cf72ebd0c8a (127.20.120.1). New cstate: current_term: 1 leader_uuid: "9772a83eea10458f835e5cf72ebd0c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } health_report { overall_health: UNKNOWN } } }
I20250814 07:57:39.079958 20960 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250814 07:57:39.285662 20960 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250814 07:57:39.422864 22349 consensus_queue.cc:1035] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 07:57:39.437229 22349 consensus_queue.cc:1035] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 07:57:41.187841 22013 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4aab946a844c42739e52573e8320599a"
dest_uuid: "9772a83eea10458f835e5cf72ebd0c8a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54264
I20250814 07:57:41.188438 22013 raft_consensus.cc:604] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 1 LEADER]: Received request to transfer leadership
I20250814 07:57:41.392674 22387 raft_consensus.cc:991] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a: : Instructing follower 9ec20e75928d47a99878f024adc93ed6 to start an election
I20250814 07:57:41.393203 22370 raft_consensus.cc:1079] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 1 LEADER]: Signalling peer 9ec20e75928d47a99878f024adc93ed6 to start an election
I20250814 07:57:41.394580 22279 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "4aab946a844c42739e52573e8320599a"
dest_uuid: "9ec20e75928d47a99878f024adc93ed6"
from {username='slave'} at 127.20.120.1:60543
I20250814 07:57:41.395061 22279 raft_consensus.cc:491] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250814 07:57:41.395306 22279 raft_consensus.cc:3058] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [term 1 FOLLOWER]: Advancing to term 2
I20250814 07:57:41.399307 22279 raft_consensus.cc:513] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:41.401566 22279 leader_election.cc:290] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [CANDIDATE]: Term 2 election: Requested vote from peers 9772a83eea10458f835e5cf72ebd0c8a (127.20.120.1:36635), 95fd1736feca440c90bdd3e9dcd04340 (127.20.120.2:34701)
I20250814 07:57:41.412299 22013 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4aab946a844c42739e52573e8320599a" candidate_uuid: "9ec20e75928d47a99878f024adc93ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "9772a83eea10458f835e5cf72ebd0c8a"
I20250814 07:57:41.412873 22013 raft_consensus.cc:3053] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 1 LEADER]: Stepping down as leader of term 1
I20250814 07:57:41.413125 22013 raft_consensus.cc:738] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 9772a83eea10458f835e5cf72ebd0c8a, State: Running, Role: LEADER
I20250814 07:57:41.413031 22146 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4aab946a844c42739e52573e8320599a" candidate_uuid: "9ec20e75928d47a99878f024adc93ed6" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "95fd1736feca440c90bdd3e9dcd04340"
I20250814 07:57:41.413443 22146 raft_consensus.cc:3058] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340 [term 1 FOLLOWER]: Advancing to term 2
I20250814 07:57:41.413618 22013 consensus_queue.cc:260] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:41.414491 22013 raft_consensus.cc:3058] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 1 FOLLOWER]: Advancing to term 2
I20250814 07:57:41.417259 22146 raft_consensus.cc:2466] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ec20e75928d47a99878f024adc93ed6 in term 2.
I20250814 07:57:41.418083 22215 leader_election.cc:304] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95fd1736feca440c90bdd3e9dcd04340, 9ec20e75928d47a99878f024adc93ed6; no voters:
I20250814 07:57:41.418500 22013 raft_consensus.cc:2466] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ec20e75928d47a99878f024adc93ed6 in term 2.
I20250814 07:57:41.420120 22391 raft_consensus.cc:2802] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 07:57:41.421360 22391 raft_consensus.cc:695] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [term 2 LEADER]: Becoming Leader. State: Replica: 9ec20e75928d47a99878f024adc93ed6, State: Running, Role: LEADER
I20250814 07:57:41.422132 22391 consensus_queue.cc:237] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } }
I20250814 07:57:41.429430 21870 catalog_manager.cc:5582] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 reported cstate change: term changed from 1 to 2, leader changed from 9772a83eea10458f835e5cf72ebd0c8a (127.20.120.1) to 9ec20e75928d47a99878f024adc93ed6 (127.20.120.3). New cstate: current_term: 2 leader_uuid: "9ec20e75928d47a99878f024adc93ed6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9ec20e75928d47a99878f024adc93ed6" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 46151 } health_report { overall_health: HEALTHY } } }
I20250814 07:57:41.997586 22013 raft_consensus.cc:1273] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 2 FOLLOWER]: Refusing update from remote peer 9ec20e75928d47a99878f024adc93ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250814 07:57:41.998682 22398 consensus_queue.cc:1035] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9772a83eea10458f835e5cf72ebd0c8a" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 36635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250814 07:57:42.008258 22146 raft_consensus.cc:1273] T 4aab946a844c42739e52573e8320599a P 95fd1736feca440c90bdd3e9dcd04340 [term 2 FOLLOWER]: Refusing update from remote peer 9ec20e75928d47a99878f024adc93ed6: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250814 07:57:42.010138 22401 consensus_queue.cc:1035] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "95fd1736feca440c90bdd3e9dcd04340" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20250814 07:57:43.921895 22013 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "4aab946a844c42739e52573e8320599a"
dest_uuid: "9772a83eea10458f835e5cf72ebd0c8a"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54272
I20250814 07:57:43.922437 22013 raft_consensus.cc:604] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 2 FOLLOWER]: Received request to transfer leadership
I20250814 07:57:43.922721 22013 raft_consensus.cc:612] T 4aab946a844c42739e52573e8320599a P 9772a83eea10458f835e5cf72ebd0c8a [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250814 07:57:44.955507 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 21929
I20250814 07:57:44.979416 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 22062
W20250814 07:57:44.993709 22213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.1:36635: connect: Connection refused (error 111)
W20250814 07:57:44.996728 22213 consensus_peers.cc:489] T 4aab946a844c42739e52573e8320599a P 9ec20e75928d47a99878f024adc93ed6 -> Peer 9772a83eea10458f835e5cf72ebd0c8a (127.20.120.1:36635): Couldn't send request to peer 9772a83eea10458f835e5cf72ebd0c8a. Status: Network error: Client connection negotiation failed: client connection to 127.20.120.1:36635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250814 07:57:45.006808 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 22195
I20250814 07:57:45.032302 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 21837
2025-08-14T07:57:45Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (14186 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250814 07:57:45.085842 20960 test_util.cc:276] Using random seed: -1595261064
I20250814 07:57:45.089666 20960 ts_itest-base.cc:115] Starting cluster with:
I20250814 07:57:45.089820 20960 ts_itest-base.cc:116] --------------
I20250814 07:57:45.089967 20960 ts_itest-base.cc:117] 3 tablet servers
I20250814 07:57:45.090097 20960 ts_itest-base.cc:118] 3 replicas per TS
I20250814 07:57:45.090227 20960 ts_itest-base.cc:119] --------------
2025-08-14T07:57:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T07:57:45Z Disabled control of system clock
I20250814 07:57:45.123801 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:41413
--webserver_interface=127.20.120.62
--webserver_port=0
--builtin_ntp_servers=127.20.120.20:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:41413 with env {}
W20250814 07:57:45.403213 22440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:45.403707 22440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:45.404104 22440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:45.434660 22440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:57:45.434918 22440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:45.435110 22440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:57:45.435294 22440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:57:45.467794 22440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:41413
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:41413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:45.468914 22440 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:45.470361 22440 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:45.479799 22446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:45.481420 22447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:45.484560 22449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:46.595789 22448 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250814 07:57:46.595837 22440 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:46.599408 22440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:46.601925 22440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:46.603282 22440 hybrid_clock.cc:648] HybridClock initialized: now 1755158266603247 us; error 39 us; skew 500 ppm
I20250814 07:57:46.604099 22440 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:46.610494 22440 webserver.cc:480] Webserver started at http://127.20.120.62:33141/ using document root <none> and password file <none>
I20250814 07:57:46.611375 22440 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:46.611594 22440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:46.612038 22440 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:46.616345 22440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "16ed55d5177c4d9382b9cff6e38af60e"
format_stamp: "Formatted at 2025-08-14 07:57:46 on dist-test-slave-87c1"
I20250814 07:57:46.617446 22440 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "16ed55d5177c4d9382b9cff6e38af60e"
format_stamp: "Formatted at 2025-08-14 07:57:46 on dist-test-slave-87c1"
I20250814 07:57:46.624135 22440 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250814 07:57:46.629233 22456 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:46.630105 22440 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250814 07:57:46.630390 22440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "16ed55d5177c4d9382b9cff6e38af60e"
format_stamp: "Formatted at 2025-08-14 07:57:46 on dist-test-slave-87c1"
I20250814 07:57:46.630690 22440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:46.678665 22440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:46.679957 22440 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:46.680323 22440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:46.747613 22440 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:41413
I20250814 07:57:46.747690 22507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:41413 every 8 connection(s)
I20250814 07:57:46.750277 22440 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 07:57:46.755090 22508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:46.760504 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 22440
I20250814 07:57:46.760876 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 07:57:46.772084 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e: Bootstrap starting.
I20250814 07:57:46.778425 22508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:46.780205 22508 log.cc:826] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:46.784984 22508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e: No bootstrap required, opened a new log
I20250814 07:57:46.802101 22508 raft_consensus.cc:357] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed55d5177c4d9382b9cff6e38af60e" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41413 } }
I20250814 07:57:46.802924 22508 raft_consensus.cc:383] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:46.803170 22508 raft_consensus.cc:738] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ed55d5177c4d9382b9cff6e38af60e, State: Initialized, Role: FOLLOWER
I20250814 07:57:46.803818 22508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed55d5177c4d9382b9cff6e38af60e" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41413 } }
I20250814 07:57:46.804311 22508 raft_consensus.cc:397] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:57:46.804592 22508 raft_consensus.cc:491] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:57:46.804893 22508 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:46.808710 22508 raft_consensus.cc:513] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed55d5177c4d9382b9cff6e38af60e" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41413 } }
I20250814 07:57:46.809509 22508 leader_election.cc:304] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16ed55d5177c4d9382b9cff6e38af60e; no voters:
I20250814 07:57:46.811110 22508 leader_election.cc:290] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:57:46.811759 22513 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:57:46.813839 22513 raft_consensus.cc:695] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [term 1 LEADER]: Becoming Leader. State: Replica: 16ed55d5177c4d9382b9cff6e38af60e, State: Running, Role: LEADER
I20250814 07:57:46.814461 22513 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed55d5177c4d9382b9cff6e38af60e" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41413 } }
I20250814 07:57:46.815024 22508 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:57:46.824749 22515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ed55d5177c4d9382b9cff6e38af60e. Latest consensus state: current_term: 1 leader_uuid: "16ed55d5177c4d9382b9cff6e38af60e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed55d5177c4d9382b9cff6e38af60e" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41413 } } }
I20250814 07:57:46.824980 22514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16ed55d5177c4d9382b9cff6e38af60e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ed55d5177c4d9382b9cff6e38af60e" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41413 } } }
I20250814 07:57:46.825727 22515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [sys.catalog]: This master's current role is: LEADER
I20250814 07:57:46.825827 22514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e [sys.catalog]: This master's current role is: LEADER
I20250814 07:57:46.829706 22522 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:57:46.840843 22522 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:57:46.857728 22522 catalog_manager.cc:1349] Generated new cluster ID: 7f5b44391eef4aaa95ded71c238a5862
I20250814 07:57:46.857947 22522 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:57:46.882330 22522 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 07:57:46.883605 22522 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:57:46.894210 22522 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 16ed55d5177c4d9382b9cff6e38af60e: Generated new TSK 0
I20250814 07:57:46.894964 22522 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 07:57:46.914433 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:0
--local_ip_for_outbound_sockets=127.20.120.1
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41413
--builtin_ntp_servers=127.20.120.20:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250814 07:57:47.211095 22532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:47.211552 22532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:47.212013 22532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:47.242535 22532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:47.243366 22532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 07:57:47.276597 22532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41413
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:47.277961 22532 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:47.279443 22532 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:47.294261 22538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:48.695837 22537 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 22532
W20250814 07:57:48.754060 22532 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.460s user 0.488s sys 0.973s
W20250814 07:57:47.295814 22539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:48.754434 22532 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.461s user 0.488s sys 0.973s
W20250814 07:57:48.763062 22543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:48.763192 22532 server_base.cc:1047] running on GCE node
I20250814 07:57:48.764238 22532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:48.766196 22532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:48.767525 22532 hybrid_clock.cc:648] HybridClock initialized: now 1755158268767471 us; error 55 us; skew 500 ppm
I20250814 07:57:48.768311 22532 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:48.773923 22532 webserver.cc:480] Webserver started at http://127.20.120.1:34417/ using document root <none> and password file <none>
I20250814 07:57:48.774804 22532 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:48.775010 22532 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:48.775434 22532 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:48.779596 22532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "970ee903db024544911cdcf50efdd345"
format_stamp: "Formatted at 2025-08-14 07:57:48 on dist-test-slave-87c1"
I20250814 07:57:48.780694 22532 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "970ee903db024544911cdcf50efdd345"
format_stamp: "Formatted at 2025-08-14 07:57:48 on dist-test-slave-87c1"
I20250814 07:57:48.787159 22532 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20250814 07:57:48.792205 22548 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:48.793112 22532 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250814 07:57:48.793398 22532 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "970ee903db024544911cdcf50efdd345"
format_stamp: "Formatted at 2025-08-14 07:57:48 on dist-test-slave-87c1"
I20250814 07:57:48.793704 22532 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:48.848508 22532 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:48.849819 22532 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:48.850239 22532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:48.852396 22532 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:57:48.856112 22532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:57:48.856300 22532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:48.856534 22532 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:57:48.856683 22532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:48.984036 22532 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:37391
I20250814 07:57:48.984135 22660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:37391 every 8 connection(s)
I20250814 07:57:48.986485 22532 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 07:57:48.987680 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 22532
I20250814 07:57:48.988082 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 07:57:48.995189 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.2:0
--local_ip_for_outbound_sockets=127.20.120.2
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41413
--builtin_ntp_servers=127.20.120.20:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:57:49.015730 22661 heartbeater.cc:344] Connected to a master server at 127.20.120.62:41413
I20250814 07:57:49.016180 22661 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:49.017326 22661 heartbeater.cc:507] Master 127.20.120.62:41413 requested a full tablet report, sending...
I20250814 07:57:49.019778 22473 ts_manager.cc:194] Registered new tserver with Master: 970ee903db024544911cdcf50efdd345 (127.20.120.1:37391)
I20250814 07:57:49.021657 22473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:41301
W20250814 07:57:49.281309 22665 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:49.281730 22665 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:49.282181 22665 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:49.311465 22665 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:49.312260 22665 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.2
I20250814 07:57:49.344823 22665 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41413
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.2
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:49.346073 22665 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:49.347481 22665 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:49.358871 22671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:50.025235 22661 heartbeater.cc:499] Master 127.20.120.62:41413 was elected leader, sending a full tablet report...
W20250814 07:57:49.359464 22672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:50.530965 22673 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250814 07:57:50.532728 22674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:50.535820 22665 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.177s user 0.371s sys 0.804s
W20250814 07:57:50.536056 22665 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.177s user 0.371s sys 0.804s
I20250814 07:57:50.536237 22665 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:50.537261 22665 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:50.539445 22665 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:50.540771 22665 hybrid_clock.cc:648] HybridClock initialized: now 1755158270540738 us; error 36 us; skew 500 ppm
I20250814 07:57:50.541548 22665 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:50.548015 22665 webserver.cc:480] Webserver started at http://127.20.120.2:40881/ using document root <none> and password file <none>
I20250814 07:57:50.548952 22665 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:50.549216 22665 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:50.549712 22665 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:50.553901 22665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "536d802144904487b5591cdde4932b57"
format_stamp: "Formatted at 2025-08-14 07:57:50 on dist-test-slave-87c1"
I20250814 07:57:50.554906 22665 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "536d802144904487b5591cdde4932b57"
format_stamp: "Formatted at 2025-08-14 07:57:50 on dist-test-slave-87c1"
I20250814 07:57:50.562377 22665 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250814 07:57:50.568580 22682 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:50.569788 22665 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250814 07:57:50.570071 22665 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "536d802144904487b5591cdde4932b57"
format_stamp: "Formatted at 2025-08-14 07:57:50 on dist-test-slave-87c1"
I20250814 07:57:50.570384 22665 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:50.647759 22665 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:50.649173 22665 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:50.649583 22665 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:50.651954 22665 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:57:50.655678 22665 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:57:50.655871 22665 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:50.656114 22665 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:57:50.656266 22665 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:50.785107 22665 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.2:34305
I20250814 07:57:50.785190 22794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.2:34305 every 8 connection(s)
I20250814 07:57:50.787528 22665 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 07:57:50.791080 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 22665
I20250814 07:57:50.791743 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 07:57:50.798475 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.3:0
--local_ip_for_outbound_sockets=127.20.120.3
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41413
--builtin_ntp_servers=127.20.120.20:34771
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:57:50.807185 22795 heartbeater.cc:344] Connected to a master server at 127.20.120.62:41413
I20250814 07:57:50.807560 22795 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:50.808424 22795 heartbeater.cc:507] Master 127.20.120.62:41413 requested a full tablet report, sending...
I20250814 07:57:50.810416 22473 ts_manager.cc:194] Registered new tserver with Master: 536d802144904487b5591cdde4932b57 (127.20.120.2:34305)
I20250814 07:57:50.811599 22473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.2:50967
W20250814 07:57:51.089058 22799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:51.089524 22799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:57:51.090011 22799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:51.119742 22799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:57:51.120527 22799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.3
I20250814 07:57:51.152921 22799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:34771
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41413
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.3
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:57:51.154129 22799 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:57:51.155625 22799 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:57:51.166728 22805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:57:51.814181 22795 heartbeater.cc:499] Master 127.20.120.62:41413 was elected leader, sending a full tablet report...
W20250814 07:57:52.572268 22804 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 22799
W20250814 07:57:52.855713 22799 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.688s user 0.618s sys 1.068s
W20250814 07:57:52.856119 22799 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.689s user 0.618s sys 1.068s
W20250814 07:57:51.167737 22806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:52.857796 22808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:57:52.860729 22807 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1690 milliseconds
I20250814 07:57:52.860755 22799 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:57:52.861918 22799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:57:52.863888 22799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:57:52.865216 22799 hybrid_clock.cc:648] HybridClock initialized: now 1755158272865171 us; error 41 us; skew 500 ppm
I20250814 07:57:52.865958 22799 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:57:52.871582 22799 webserver.cc:480] Webserver started at http://127.20.120.3:40459/ using document root <none> and password file <none>
I20250814 07:57:52.872442 22799 fs_manager.cc:362] Metadata directory not provided
I20250814 07:57:52.872651 22799 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:57:52.873106 22799 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:57:52.877377 22799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7ad6820db73048c79a4723b729c18f0e"
format_stamp: "Formatted at 2025-08-14 07:57:52 on dist-test-slave-87c1"
I20250814 07:57:52.878412 22799 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7ad6820db73048c79a4723b729c18f0e"
format_stamp: "Formatted at 2025-08-14 07:57:52 on dist-test-slave-87c1"
I20250814 07:57:52.885051 22799 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20250814 07:57:52.890170 22815 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:52.891077 22799 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.005s
I20250814 07:57:52.891386 22799 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7ad6820db73048c79a4723b729c18f0e"
format_stamp: "Formatted at 2025-08-14 07:57:52 on dist-test-slave-87c1"
I20250814 07:57:52.891682 22799 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:57:52.939443 22799 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:57:52.940793 22799 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:57:52.941212 22799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:57:52.943560 22799 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:57:52.947245 22799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:57:52.947453 22799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:52.947686 22799 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:57:52.947841 22799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:57:53.085346 22799 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.3:39765
I20250814 07:57:53.085454 22927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.3:39765 every 8 connection(s)
I20250814 07:57:53.087841 22799 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 07:57:53.093451 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 22799
I20250814 07:57:53.093954 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 07:57:53.107673 22928 heartbeater.cc:344] Connected to a master server at 127.20.120.62:41413
I20250814 07:57:53.108049 22928 heartbeater.cc:461] Registering TS with master...
I20250814 07:57:53.108981 22928 heartbeater.cc:507] Master 127.20.120.62:41413 requested a full tablet report, sending...
I20250814 07:57:53.110966 22473 ts_manager.cc:194] Registered new tserver with Master: 7ad6820db73048c79a4723b729c18f0e (127.20.120.3:39765)
I20250814 07:57:53.112289 22473 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.3:56555
I20250814 07:57:53.113648 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:57:53.143927 22472 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51066:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250814 07:57:53.161644 22472 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 07:57:53.206796 22730 tablet_service.cc:1468] Processing CreateTablet for tablet 9d675450b08d48578dc73fa6b1071e2e (DEFAULT_TABLE table=TestTable [id=216eff74f5b24398bbd00154d1ddbae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:57:53.209331 22730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d675450b08d48578dc73fa6b1071e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:53.212798 22863 tablet_service.cc:1468] Processing CreateTablet for tablet 9d675450b08d48578dc73fa6b1071e2e (DEFAULT_TABLE table=TestTable [id=216eff74f5b24398bbd00154d1ddbae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:57:53.214737 22863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d675450b08d48578dc73fa6b1071e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:53.215036 22596 tablet_service.cc:1468] Processing CreateTablet for tablet 9d675450b08d48578dc73fa6b1071e2e (DEFAULT_TABLE table=TestTable [id=216eff74f5b24398bbd00154d1ddbae6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:57:53.216544 22596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d675450b08d48578dc73fa6b1071e2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:53.236753 22947 tablet_bootstrap.cc:492] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345: Bootstrap starting.
I20250814 07:57:53.238225 22948 tablet_bootstrap.cc:492] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57: Bootstrap starting.
I20250814 07:57:53.240983 22949 tablet_bootstrap.cc:492] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e: Bootstrap starting.
I20250814 07:57:53.244369 22947 tablet_bootstrap.cc:654] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:53.245821 22948 tablet_bootstrap.cc:654] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:53.246677 22947 log.cc:826] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:53.247471 22949 tablet_bootstrap.cc:654] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:53.248102 22948 log.cc:826] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:53.249459 22949 log.cc:826] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e: Log is configured to *not* fsync() on all Append() calls
I20250814 07:57:53.252590 22947 tablet_bootstrap.cc:492] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345: No bootstrap required, opened a new log
I20250814 07:57:53.253063 22947 ts_tablet_manager.cc:1397] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.004s
I20250814 07:57:53.254623 22948 tablet_bootstrap.cc:492] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57: No bootstrap required, opened a new log
I20250814 07:57:53.254784 22949 tablet_bootstrap.cc:492] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e: No bootstrap required, opened a new log
I20250814 07:57:53.255005 22948 ts_tablet_manager.cc:1397] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.008s
I20250814 07:57:53.255223 22949 ts_tablet_manager.cc:1397] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.006s
I20250814 07:57:53.272601 22948 raft_consensus.cc:357] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.273300 22948 raft_consensus.cc:383] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:53.273584 22948 raft_consensus.cc:738] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536d802144904487b5591cdde4932b57, State: Initialized, Role: FOLLOWER
I20250814 07:57:53.274256 22948 consensus_queue.cc:260] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.279141 22947 raft_consensus.cc:357] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.279839 22948 ts_tablet_manager.cc:1428] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57: Time spent starting tablet: real 0.025s user 0.017s sys 0.009s
I20250814 07:57:53.279997 22947 raft_consensus.cc:383] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:53.280301 22947 raft_consensus.cc:738] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970ee903db024544911cdcf50efdd345, State: Initialized, Role: FOLLOWER
I20250814 07:57:53.280987 22949 raft_consensus.cc:357] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.281163 22947 consensus_queue.cc:260] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.281836 22949 raft_consensus.cc:383] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:53.282111 22949 raft_consensus.cc:738] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad6820db73048c79a4723b729c18f0e, State: Initialized, Role: FOLLOWER
I20250814 07:57:53.283026 22949 consensus_queue.cc:260] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.287532 22928 heartbeater.cc:499] Master 127.20.120.62:41413 was elected leader, sending a full tablet report...
I20250814 07:57:53.289350 22949 ts_tablet_manager.cc:1428] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e: Time spent starting tablet: real 0.034s user 0.023s sys 0.013s
I20250814 07:57:53.291110 22947 ts_tablet_manager.cc:1428] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345: Time spent starting tablet: real 0.038s user 0.027s sys 0.009s
W20250814 07:57:53.294107 22796 tablet.cc:2378] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 07:57:53.342224 22929 tablet.cc:2378] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:57:53.384166 22953 raft_consensus.cc:491] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:57:53.384714 22953 raft_consensus.cc:513] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.387251 22953 leader_election.cc:290] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 970ee903db024544911cdcf50efdd345 (127.20.120.1:37391), 7ad6820db73048c79a4723b729c18f0e (127.20.120.3:39765)
I20250814 07:57:53.398918 22616 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d675450b08d48578dc73fa6b1071e2e" candidate_uuid: "536d802144904487b5591cdde4932b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970ee903db024544911cdcf50efdd345" is_pre_election: true
I20250814 07:57:53.399686 22616 raft_consensus.cc:2466] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 536d802144904487b5591cdde4932b57 in term 0.
I20250814 07:57:53.400692 22883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d675450b08d48578dc73fa6b1071e2e" candidate_uuid: "536d802144904487b5591cdde4932b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad6820db73048c79a4723b729c18f0e" is_pre_election: true
I20250814 07:57:53.400785 22685 leader_election.cc:304] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 536d802144904487b5591cdde4932b57, 970ee903db024544911cdcf50efdd345; no voters:
I20250814 07:57:53.401352 22883 raft_consensus.cc:2466] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 536d802144904487b5591cdde4932b57 in term 0.
I20250814 07:57:53.401556 22953 raft_consensus.cc:2802] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 07:57:53.401882 22953 raft_consensus.cc:491] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 07:57:53.402211 22953 raft_consensus.cc:3058] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:53.406531 22953 raft_consensus.cc:513] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.407910 22953 leader_election.cc:290] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [CANDIDATE]: Term 1 election: Requested vote from peers 970ee903db024544911cdcf50efdd345 (127.20.120.1:37391), 7ad6820db73048c79a4723b729c18f0e (127.20.120.3:39765)
I20250814 07:57:53.408596 22616 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d675450b08d48578dc73fa6b1071e2e" candidate_uuid: "536d802144904487b5591cdde4932b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970ee903db024544911cdcf50efdd345"
I20250814 07:57:53.408962 22883 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d675450b08d48578dc73fa6b1071e2e" candidate_uuid: "536d802144904487b5591cdde4932b57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ad6820db73048c79a4723b729c18f0e"
I20250814 07:57:53.409111 22616 raft_consensus.cc:3058] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:53.409487 22883 raft_consensus.cc:3058] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:53.416062 22883 raft_consensus.cc:2466] T 9d675450b08d48578dc73fa6b1071e2e P 7ad6820db73048c79a4723b729c18f0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 536d802144904487b5591cdde4932b57 in term 1.
I20250814 07:57:53.416061 22616 raft_consensus.cc:2466] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 536d802144904487b5591cdde4932b57 in term 1.
I20250814 07:57:53.416934 22685 leader_election.cc:304] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 536d802144904487b5591cdde4932b57, 970ee903db024544911cdcf50efdd345; no voters:
I20250814 07:57:53.417510 22953 raft_consensus.cc:2802] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:57:53.419078 22953 raft_consensus.cc:695] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [term 1 LEADER]: Becoming Leader. State: Replica: 536d802144904487b5591cdde4932b57, State: Running, Role: LEADER
I20250814 07:57:53.419894 22953 consensus_queue.cc:237] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.428709 22473 catalog_manager.cc:5582] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 reported cstate change: term changed from 0 to 1, leader changed from <none> to 536d802144904487b5591cdde4932b57 (127.20.120.2). New cstate: current_term: 1 leader_uuid: "536d802144904487b5591cdde4932b57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } health_report { overall_health: UNKNOWN } } }
I20250814 07:57:53.446868 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:57:53.449857 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 970ee903db024544911cdcf50efdd345 to finish bootstrapping
I20250814 07:57:53.461319 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 536d802144904487b5591cdde4932b57 to finish bootstrapping
I20250814 07:57:53.471427 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7ad6820db73048c79a4723b729c18f0e to finish bootstrapping
I20250814 07:57:53.483808 22473 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51066:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250814 07:57:53.485282 22473 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250814 07:57:53.500305 22662 tablet.cc:2378] T 9d675450b08d48578dc73fa6b1071e2e P 970ee903db024544911cdcf50efdd345: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:57:53.500372 22596 tablet_service.cc:1468] Processing CreateTablet for tablet 253b42ba26334c4cbe923bf10b21a728 (DEFAULT_TABLE table=TestAnotherTable [id=1ed40fd9c9af4c30a06a213061c35e42]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250814 07:57:53.501029 22730 tablet_service.cc:1468] Processing CreateTablet for tablet 253b42ba26334c4cbe923bf10b21a728 (DEFAULT_TABLE table=TestAnotherTable [id=1ed40fd9c9af4c30a06a213061c35e42]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250814 07:57:53.501494 22596 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 253b42ba26334c4cbe923bf10b21a728. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:53.502060 22730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 253b42ba26334c4cbe923bf10b21a728. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:53.502888 22863 tablet_service.cc:1468] Processing CreateTablet for tablet 253b42ba26334c4cbe923bf10b21a728 (DEFAULT_TABLE table=TestAnotherTable [id=1ed40fd9c9af4c30a06a213061c35e42]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250814 07:57:53.503659 22863 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 253b42ba26334c4cbe923bf10b21a728. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:57:53.513419 22949 tablet_bootstrap.cc:492] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e: Bootstrap starting.
I20250814 07:57:53.515472 22948 tablet_bootstrap.cc:492] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57: Bootstrap starting.
I20250814 07:57:53.515486 22947 tablet_bootstrap.cc:492] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345: Bootstrap starting.
I20250814 07:57:53.519351 22949 tablet_bootstrap.cc:654] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:53.521385 22948 tablet_bootstrap.cc:654] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:53.521565 22947 tablet_bootstrap.cc:654] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345: Neither blocks nor log segments found. Creating new log.
I20250814 07:57:53.524987 22949 tablet_bootstrap.cc:492] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e: No bootstrap required, opened a new log
I20250814 07:57:53.525735 22949 ts_tablet_manager.cc:1397] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20250814 07:57:53.528966 22949 raft_consensus.cc:357] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.529670 22949 raft_consensus.cc:383] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:53.529950 22949 raft_consensus.cc:738] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ad6820db73048c79a4723b729c18f0e, State: Initialized, Role: FOLLOWER
I20250814 07:57:53.530476 22948 tablet_bootstrap.cc:492] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57: No bootstrap required, opened a new log
I20250814 07:57:53.530877 22948 ts_tablet_manager.cc:1397] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20250814 07:57:53.531020 22947 tablet_bootstrap.cc:492] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345: No bootstrap required, opened a new log
I20250814 07:57:53.530608 22949 consensus_queue.cc:260] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.531396 22947 ts_tablet_manager.cc:1397] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250814 07:57:53.532768 22949 ts_tablet_manager.cc:1428] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e: Time spent starting tablet: real 0.007s user 0.001s sys 0.003s
I20250814 07:57:53.533419 22948 raft_consensus.cc:357] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.534070 22948 raft_consensus.cc:383] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:53.534372 22948 raft_consensus.cc:738] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 536d802144904487b5591cdde4932b57, State: Initialized, Role: FOLLOWER
I20250814 07:57:53.534425 22947 raft_consensus.cc:357] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.535074 22947 raft_consensus.cc:383] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:57:53.535348 22947 raft_consensus.cc:738] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970ee903db024544911cdcf50efdd345, State: Initialized, Role: FOLLOWER
I20250814 07:57:53.535073 22948 consensus_queue.cc:260] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.536070 22947 consensus_queue.cc:260] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.538395 22947 ts_tablet_manager.cc:1428] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345: Time spent starting tablet: real 0.007s user 0.004s sys 0.001s
I20250814 07:57:53.538679 22948 ts_tablet_manager.cc:1428] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20250814 07:57:53.744400 22955 raft_consensus.cc:491] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:57:53.744781 22955 raft_consensus.cc:513] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.746855 22955 leader_election.cc:290] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 970ee903db024544911cdcf50efdd345 (127.20.120.1:37391), 536d802144904487b5591cdde4932b57 (127.20.120.2:34305)
I20250814 07:57:53.757979 22616 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "253b42ba26334c4cbe923bf10b21a728" candidate_uuid: "7ad6820db73048c79a4723b729c18f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970ee903db024544911cdcf50efdd345" is_pre_election: true
I20250814 07:57:53.758533 22616 raft_consensus.cc:2466] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ad6820db73048c79a4723b729c18f0e in term 0.
I20250814 07:57:53.759572 22818 leader_election.cc:304] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ad6820db73048c79a4723b729c18f0e, 970ee903db024544911cdcf50efdd345; no voters:
I20250814 07:57:53.760363 22955 raft_consensus.cc:2802] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 07:57:53.760673 22955 raft_consensus.cc:491] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 07:57:53.760943 22955 raft_consensus.cc:3058] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:53.762394 22750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "253b42ba26334c4cbe923bf10b21a728" candidate_uuid: "7ad6820db73048c79a4723b729c18f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536d802144904487b5591cdde4932b57" is_pre_election: true
I20250814 07:57:53.762959 22750 raft_consensus.cc:2466] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7ad6820db73048c79a4723b729c18f0e in term 0.
I20250814 07:57:53.767022 22955 raft_consensus.cc:513] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.768602 22955 leader_election.cc:290] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [CANDIDATE]: Term 1 election: Requested vote from peers 970ee903db024544911cdcf50efdd345 (127.20.120.1:37391), 536d802144904487b5591cdde4932b57 (127.20.120.2:34305)
I20250814 07:57:53.769341 22616 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "253b42ba26334c4cbe923bf10b21a728" candidate_uuid: "7ad6820db73048c79a4723b729c18f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "970ee903db024544911cdcf50efdd345"
I20250814 07:57:53.769507 22750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "253b42ba26334c4cbe923bf10b21a728" candidate_uuid: "7ad6820db73048c79a4723b729c18f0e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "536d802144904487b5591cdde4932b57"
I20250814 07:57:53.769800 22616 raft_consensus.cc:3058] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:53.770020 22750 raft_consensus.cc:3058] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:57:53.773828 22616 raft_consensus.cc:2466] T 253b42ba26334c4cbe923bf10b21a728 P 970ee903db024544911cdcf50efdd345 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ad6820db73048c79a4723b729c18f0e in term 1.
I20250814 07:57:53.774572 22818 leader_election.cc:304] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7ad6820db73048c79a4723b729c18f0e, 970ee903db024544911cdcf50efdd345; no voters:
I20250814 07:57:53.775128 22955 raft_consensus.cc:2802] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:57:53.776079 22750 raft_consensus.cc:2466] T 253b42ba26334c4cbe923bf10b21a728 P 536d802144904487b5591cdde4932b57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7ad6820db73048c79a4723b729c18f0e in term 1.
I20250814 07:57:53.777828 22955 raft_consensus.cc:695] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [term 1 LEADER]: Becoming Leader. State: Replica: 7ad6820db73048c79a4723b729c18f0e, State: Running, Role: LEADER
I20250814 07:57:53.778640 22955 consensus_queue.cc:237] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } } peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } }
I20250814 07:57:53.790731 22471 catalog_manager.cc:5582] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e reported cstate change: term changed from 0 to 1, leader changed from <none> to 7ad6820db73048c79a4723b729c18f0e (127.20.120.3). New cstate: current_term: 1 leader_uuid: "7ad6820db73048c79a4723b729c18f0e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 } health_report { overall_health: HEALTHY } } }
I20250814 07:57:53.826117 22953 consensus_queue.cc:1035] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 07:57:53.850781 22953 consensus_queue.cc:1035] T 9d675450b08d48578dc73fa6b1071e2e P 536d802144904487b5591cdde4932b57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ad6820db73048c79a4723b729c18f0e" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250814 07:57:54.120288 22968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:54.120822 22968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:54.149356 22968 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250814 07:57:54.316129 22955 consensus_queue.cc:1035] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "970ee903db024544911cdcf50efdd345" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 37391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250814 07:57:54.340209 22986 consensus_queue.cc:1035] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e [LEADER]: Connected to new peer: Peer: permanent_uuid: "536d802144904487b5591cdde4932b57" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 34305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250814 07:57:55.416723 22968 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.232s user 0.436s sys 0.791s
W20250814 07:57:55.417174 22968 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.232s user 0.439s sys 0.791s
W20250814 07:57:56.790805 22995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:56.791330 22995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:56.820858 22995 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250814 07:57:58.058316 22995 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.201s user 0.538s sys 0.662s
W20250814 07:57:58.058629 22995 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.201s user 0.538s sys 0.662s
W20250814 07:57:59.425923 23009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:57:59.426443 23009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:57:59.455974 23009 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250814 07:58:00.690351 23009 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.194s user 0.418s sys 0.774s
W20250814 07:58:00.690611 23009 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.195s user 0.418s sys 0.774s
W20250814 07:58:02.063639 23027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:02.064213 23027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:02.093114 23027 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250814 07:58:03.314652 23027 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.184s user 0.425s sys 0.758s
W20250814 07:58:03.315047 23027 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.185s user 0.425s sys 0.758s
I20250814 07:58:04.388156 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 22532
I20250814 07:58:04.413169 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 22665
W20250814 07:58:04.431064 22818 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.1:37391: connect: Connection refused (error 111)
W20250814 07:58:04.434438 22818 consensus_peers.cc:489] T 253b42ba26334c4cbe923bf10b21a728 P 7ad6820db73048c79a4723b729c18f0e -> Peer 970ee903db024544911cdcf50efdd345 (127.20.120.1:37391): Couldn't send request to peer 970ee903db024544911cdcf50efdd345. Status: Network error: Client connection negotiation failed: client connection to 127.20.120.1:37391: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250814 07:58:04.441612 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 22799
I20250814 07:58:04.469744 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 22440
2025-08-14T07:58:04Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (19440 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250814 07:58:04.526873 20960 test_util.cc:276] Using random seed: -1575820034
I20250814 07:58:04.530831 20960 ts_itest-base.cc:115] Starting cluster with:
I20250814 07:58:04.530975 20960 ts_itest-base.cc:116] --------------
I20250814 07:58:04.531143 20960 ts_itest-base.cc:117] 3 tablet servers
I20250814 07:58:04.531288 20960 ts_itest-base.cc:118] 3 replicas per TS
I20250814 07:58:04.531436 20960 ts_itest-base.cc:119] --------------
2025-08-14T07:58:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T07:58:04Z Disabled control of system clock
I20250814 07:58:04.566088 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:40367
--webserver_interface=127.20.120.62
--webserver_port=0
--builtin_ntp_servers=127.20.120.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:40367
--superuser_acl=no-such-user with env {}
W20250814 07:58:04.842728 23050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:04.843220 23050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:04.843636 23050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:04.872035 23050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:58:04.872268 23050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:04.872460 23050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:58:04.872645 23050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:58:04.908594 23050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:40367
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:40367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:04.910079 23050 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:04.911569 23050 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:04.921877 23056 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:04.922253 23057 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:06.128834 23059 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:06.133268 23058 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1209 milliseconds
W20250814 07:58:06.133412 23050 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.212s user 0.449s sys 0.755s
W20250814 07:58:06.133728 23050 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.212s user 0.449s sys 0.755s
I20250814 07:58:06.133960 23050 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:06.135022 23050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:06.137619 23050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:06.138983 23050 hybrid_clock.cc:648] HybridClock initialized: now 1755158286138927 us; error 88 us; skew 500 ppm
I20250814 07:58:06.139714 23050 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:06.146381 23050 webserver.cc:480] Webserver started at http://127.20.120.62:34321/ using document root <none> and password file <none>
I20250814 07:58:06.147416 23050 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:06.147644 23050 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:06.148084 23050 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:58:06.152303 23050 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "764628e8c4894e27ad7545945e3119ba"
format_stamp: "Formatted at 2025-08-14 07:58:06 on dist-test-slave-87c1"
I20250814 07:58:06.153381 23050 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "764628e8c4894e27ad7545945e3119ba"
format_stamp: "Formatted at 2025-08-14 07:58:06 on dist-test-slave-87c1"
I20250814 07:58:06.160742 23050 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250814 07:58:06.166647 23066 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:06.167732 23050 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250814 07:58:06.168066 23050 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "764628e8c4894e27ad7545945e3119ba"
format_stamp: "Formatted at 2025-08-14 07:58:06 on dist-test-slave-87c1"
I20250814 07:58:06.168356 23050 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:06.237854 23050 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:06.239249 23050 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:06.239688 23050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:06.306759 23050 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:40367
I20250814 07:58:06.306839 23117 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:40367 every 8 connection(s)
I20250814 07:58:06.309298 23050 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 07:58:06.312062 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 23050
I20250814 07:58:06.312618 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 07:58:06.314788 23118 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:06.338055 23118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba: Bootstrap starting.
I20250814 07:58:06.343583 23118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba: Neither blocks nor log segments found. Creating new log.
I20250814 07:58:06.345142 23118 log.cc:826] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:06.349418 23118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba: No bootstrap required, opened a new log
I20250814 07:58:06.367162 23118 raft_consensus.cc:357] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764628e8c4894e27ad7545945e3119ba" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40367 } }
I20250814 07:58:06.367722 23118 raft_consensus.cc:383] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:58:06.367909 23118 raft_consensus.cc:738] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 764628e8c4894e27ad7545945e3119ba, State: Initialized, Role: FOLLOWER
I20250814 07:58:06.368510 23118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764628e8c4894e27ad7545945e3119ba" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40367 } }
I20250814 07:58:06.368956 23118 raft_consensus.cc:397] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:06.369220 23118 raft_consensus.cc:491] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:06.369462 23118 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:58:06.373065 23118 raft_consensus.cc:513] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764628e8c4894e27ad7545945e3119ba" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40367 } }
I20250814 07:58:06.373802 23118 leader_election.cc:304] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 764628e8c4894e27ad7545945e3119ba; no voters:
I20250814 07:58:06.375614 23118 leader_election.cc:290] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:58:06.376315 23123 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:58:06.378419 23123 raft_consensus.cc:695] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [term 1 LEADER]: Becoming Leader. State: Replica: 764628e8c4894e27ad7545945e3119ba, State: Running, Role: LEADER
I20250814 07:58:06.379091 23123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764628e8c4894e27ad7545945e3119ba" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40367 } }
I20250814 07:58:06.380014 23118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:58:06.385902 23124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "764628e8c4894e27ad7545945e3119ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764628e8c4894e27ad7545945e3119ba" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40367 } } }
I20250814 07:58:06.386722 23124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [sys.catalog]: This master's current role is: LEADER
I20250814 07:58:06.386430 23125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 764628e8c4894e27ad7545945e3119ba. Latest consensus state: current_term: 1 leader_uuid: "764628e8c4894e27ad7545945e3119ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "764628e8c4894e27ad7545945e3119ba" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40367 } } }
I20250814 07:58:06.387187 23125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba [sys.catalog]: This master's current role is: LEADER
I20250814 07:58:06.391425 23130 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:58:06.403468 23130 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:58:06.418351 23130 catalog_manager.cc:1349] Generated new cluster ID: 4d3799c548524d6c9fa65baf1a9ce653
I20250814 07:58:06.418624 23130 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:58:06.459396 23130 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 07:58:06.460650 23130 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:58:06.476761 23130 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 764628e8c4894e27ad7545945e3119ba: Generated new TSK 0
I20250814 07:58:06.477507 23130 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 07:58:06.487862 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:0
--local_ip_for_outbound_sockets=127.20.120.1
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40367
--builtin_ntp_servers=127.20.120.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250814 07:58:06.773290 23142 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:06.773726 23142 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:06.774169 23142 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:06.803308 23142 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:06.804111 23142 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 07:58:06.835806 23142 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:06.836966 23142 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:06.838394 23142 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:06.851723 23149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:06.856897 23151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:08.179384 23150 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1325 milliseconds
I20250814 07:58:08.179476 23142 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250814 07:58:06.852890 23148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:08.184180 23142 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:08.186750 23142 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:08.188210 23142 hybrid_clock.cc:648] HybridClock initialized: now 1755158288188156 us; error 71 us; skew 500 ppm
I20250814 07:58:08.188974 23142 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:08.195585 23142 webserver.cc:480] Webserver started at http://127.20.120.1:46183/ using document root <none> and password file <none>
I20250814 07:58:08.196478 23142 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:08.196687 23142 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:08.197165 23142 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:58:08.201352 23142 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e407e11d3dc749af97f55ebb6d29c6df"
format_stamp: "Formatted at 2025-08-14 07:58:08 on dist-test-slave-87c1"
I20250814 07:58:08.202346 23142 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e407e11d3dc749af97f55ebb6d29c6df"
format_stamp: "Formatted at 2025-08-14 07:58:08 on dist-test-slave-87c1"
I20250814 07:58:08.209419 23142 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20250814 07:58:08.215909 23158 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:08.216912 23142 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250814 07:58:08.217242 23142 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e407e11d3dc749af97f55ebb6d29c6df"
format_stamp: "Formatted at 2025-08-14 07:58:08 on dist-test-slave-87c1"
I20250814 07:58:08.217558 23142 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:08.271674 23142 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:08.273007 23142 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:08.273451 23142 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:08.275928 23142 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:08.280077 23142 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:58:08.280279 23142 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:08.280511 23142 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:58:08.280670 23142 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:08.434744 23142 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:43623
I20250814 07:58:08.434845 23270 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:43623 every 8 connection(s)
I20250814 07:58:08.437356 23142 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 07:58:08.440349 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 23142
I20250814 07:58:08.440781 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 07:58:08.449221 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.2:0
--local_ip_for_outbound_sockets=127.20.120.2
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40367
--builtin_ntp_servers=127.20.120.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:58:08.462651 23271 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40367
I20250814 07:58:08.463039 23271 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:08.463982 23271 heartbeater.cc:507] Master 127.20.120.62:40367 requested a full tablet report, sending...
I20250814 07:58:08.466295 23083 ts_manager.cc:194] Registered new tserver with Master: e407e11d3dc749af97f55ebb6d29c6df (127.20.120.1:43623)
I20250814 07:58:08.468112 23083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:42219
W20250814 07:58:08.756479 23275 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:08.756917 23275 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:08.757398 23275 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:08.785370 23275 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:08.786191 23275 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.2
I20250814 07:58:08.817474 23275 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.2
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:08.818711 23275 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:08.820180 23275 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:08.831130 23281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:09.471203 23271 heartbeater.cc:499] Master 127.20.120.62:40367 was elected leader, sending a full tablet report...
W20250814 07:58:08.831972 23282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:10.237069 23280 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 23275
W20250814 07:58:10.307061 23275 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.475s user 0.594s sys 0.878s
W20250814 07:58:10.308889 23275 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.477s user 0.595s sys 0.878s
W20250814 07:58:10.309636 23284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:10.312036 23283 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1476 milliseconds
I20250814 07:58:10.312089 23275 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:10.313432 23275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:10.315554 23275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:10.316885 23275 hybrid_clock.cc:648] HybridClock initialized: now 1755158290316857 us; error 30 us; skew 500 ppm
I20250814 07:58:10.317663 23275 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:10.323616 23275 webserver.cc:480] Webserver started at http://127.20.120.2:45469/ using document root <none> and password file <none>
I20250814 07:58:10.324501 23275 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:10.324718 23275 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:10.325183 23275 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:58:10.329501 23275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a665ffbad1294856aacb3a511c6ab5da"
format_stamp: "Formatted at 2025-08-14 07:58:10 on dist-test-slave-87c1"
I20250814 07:58:10.330538 23275 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a665ffbad1294856aacb3a511c6ab5da"
format_stamp: "Formatted at 2025-08-14 07:58:10 on dist-test-slave-87c1"
I20250814 07:58:10.337128 23275 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250814 07:58:10.342432 23291 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:10.343317 23275 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250814 07:58:10.343627 23275 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a665ffbad1294856aacb3a511c6ab5da"
format_stamp: "Formatted at 2025-08-14 07:58:10 on dist-test-slave-87c1"
I20250814 07:58:10.343936 23275 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:10.386927 23275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:10.388237 23275 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:10.388671 23275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:10.391001 23275 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:10.395629 23275 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:58:10.395844 23275 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:10.396082 23275 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:58:10.396261 23275 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:10.530982 23275 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.2:45411
I20250814 07:58:10.531075 23404 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.2:45411 every 8 connection(s)
I20250814 07:58:10.533409 23275 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 07:58:10.541813 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 23275
I20250814 07:58:10.542361 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 07:58:10.548231 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.3:0
--local_ip_for_outbound_sockets=127.20.120.3
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40367
--builtin_ntp_servers=127.20.120.20:42575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:58:10.552615 23405 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40367
I20250814 07:58:10.552953 23405 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:10.553864 23405 heartbeater.cc:507] Master 127.20.120.62:40367 requested a full tablet report, sending...
I20250814 07:58:10.555650 23083 ts_manager.cc:194] Registered new tserver with Master: a665ffbad1294856aacb3a511c6ab5da (127.20.120.2:45411)
I20250814 07:58:10.556782 23083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.2:42613
W20250814 07:58:10.826788 23409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:10.827275 23409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:10.827746 23409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:10.856976 23409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:10.857906 23409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.3
I20250814 07:58:10.891057 23409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:42575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.3
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:10.892278 23409 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:10.893787 23409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:10.906589 23415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:11.559509 23405 heartbeater.cc:499] Master 127.20.120.62:40367 was elected leader, sending a full tablet report...
W20250814 07:58:10.907502 23416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:12.125764 23418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:12.129056 23417 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1220 milliseconds
W20250814 07:58:12.129895 23409 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.223s user 0.430s sys 0.782s
W20250814 07:58:12.130120 23409 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.223s user 0.430s sys 0.782s
I20250814 07:58:12.130338 23409 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:12.131290 23409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:12.133482 23409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:12.134820 23409 hybrid_clock.cc:648] HybridClock initialized: now 1755158292134787 us; error 36 us; skew 500 ppm
I20250814 07:58:12.135588 23409 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:12.142921 23409 webserver.cc:480] Webserver started at http://127.20.120.3:35883/ using document root <none> and password file <none>
I20250814 07:58:12.143846 23409 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:12.144043 23409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:12.144490 23409 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:58:12.148687 23409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "fbb34b845fe14c048512f2e791b9564b"
format_stamp: "Formatted at 2025-08-14 07:58:12 on dist-test-slave-87c1"
I20250814 07:58:12.149726 23409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "fbb34b845fe14c048512f2e791b9564b"
format_stamp: "Formatted at 2025-08-14 07:58:12 on dist-test-slave-87c1"
I20250814 07:58:12.157317 23409 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20250814 07:58:12.162839 23425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:12.163944 23409 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250814 07:58:12.164261 23409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "fbb34b845fe14c048512f2e791b9564b"
format_stamp: "Formatted at 2025-08-14 07:58:12 on dist-test-slave-87c1"
I20250814 07:58:12.164588 23409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:12.229410 23409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:12.230736 23409 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:12.231132 23409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:12.233531 23409 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:12.237488 23409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:58:12.237691 23409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:12.237870 23409 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:58:12.237988 23409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:12.371667 23409 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.3:41249
I20250814 07:58:12.371764 23537 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.3:41249 every 8 connection(s)
I20250814 07:58:12.374125 23409 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 07:58:12.379141 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 23409
I20250814 07:58:12.379594 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 07:58:12.398644 23538 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40367
I20250814 07:58:12.399056 23538 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:12.400060 23538 heartbeater.cc:507] Master 127.20.120.62:40367 requested a full tablet report, sending...
I20250814 07:58:12.402060 23083 ts_manager.cc:194] Registered new tserver with Master: fbb34b845fe14c048512f2e791b9564b (127.20.120.3:41249)
I20250814 07:58:12.403201 23083 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.3:44091
I20250814 07:58:12.412783 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:58:12.444093 23083 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38384:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250814 07:58:12.461337 23083 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 07:58:12.504182 23206 tablet_service.cc:1468] Processing CreateTablet for tablet a70a8c8be85b44e3bf55b46f1bdbc4f4 (DEFAULT_TABLE table=TestTable [id=71510d17fcd247c8a43e3c2fe31c6ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:58:12.506708 23206 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70a8c8be85b44e3bf55b46f1bdbc4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:12.508447 23340 tablet_service.cc:1468] Processing CreateTablet for tablet a70a8c8be85b44e3bf55b46f1bdbc4f4 (DEFAULT_TABLE table=TestTable [id=71510d17fcd247c8a43e3c2fe31c6ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:58:12.510226 23340 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70a8c8be85b44e3bf55b46f1bdbc4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:12.510141 23473 tablet_service.cc:1468] Processing CreateTablet for tablet a70a8c8be85b44e3bf55b46f1bdbc4f4 (DEFAULT_TABLE table=TestTable [id=71510d17fcd247c8a43e3c2fe31c6ea1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:58:12.511710 23473 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a70a8c8be85b44e3bf55b46f1bdbc4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:12.540231 23559 tablet_bootstrap.cc:492] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da: Bootstrap starting.
I20250814 07:58:12.544318 23557 tablet_bootstrap.cc:492] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df: Bootstrap starting.
I20250814 07:58:12.545331 23558 tablet_bootstrap.cc:492] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b: Bootstrap starting.
I20250814 07:58:12.551729 23557 tablet_bootstrap.cc:654] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df: Neither blocks nor log segments found. Creating new log.
I20250814 07:58:12.552522 23558 tablet_bootstrap.cc:654] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b: Neither blocks nor log segments found. Creating new log.
I20250814 07:58:12.554107 23557 log.cc:826] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:12.554783 23558 log.cc:826] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:12.555379 23559 tablet_bootstrap.cc:654] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da: Neither blocks nor log segments found. Creating new log.
I20250814 07:58:12.557610 23559 log.cc:826] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:12.565871 23558 tablet_bootstrap.cc:492] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b: No bootstrap required, opened a new log
I20250814 07:58:12.566447 23558 ts_tablet_manager.cc:1397] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b: Time spent bootstrapping tablet: real 0.022s user 0.013s sys 0.007s
I20250814 07:58:12.566803 23557 tablet_bootstrap.cc:492] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df: No bootstrap required, opened a new log
I20250814 07:58:12.567385 23557 ts_tablet_manager.cc:1397] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df: Time spent bootstrapping tablet: real 0.024s user 0.020s sys 0.000s
I20250814 07:58:12.574127 23559 tablet_bootstrap.cc:492] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da: No bootstrap required, opened a new log
I20250814 07:58:12.574672 23559 ts_tablet_manager.cc:1397] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da: Time spent bootstrapping tablet: real 0.035s user 0.013s sys 0.011s
I20250814 07:58:12.592339 23559 raft_consensus.cc:357] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 } } peers { permanent_uuid: "a665ffbad1294856aacb3a511c6ab5da" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 45411 } } peers { permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 } }
I20250814 07:58:12.592947 23559 raft_consensus.cc:383] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:58:12.592530 23558 raft_consensus.cc:357] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 } } peers { permanent_uuid: "a665ffbad1294856aacb3a511c6ab5da" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 45411 } } peers { permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 } }
I20250814 07:58:12.593245 23559 raft_consensus.cc:738] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a665ffbad1294856aacb3a511c6ab5da, State: Initialized, Role: FOLLOWER
I20250814 07:58:12.593382 23558 raft_consensus.cc:383] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:58:12.593691 23558 raft_consensus.cc:738] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbb34b845fe14c048512f2e791b9564b, State: Initialized, Role: FOLLOWER
I20250814 07:58:12.593393 23557 raft_consensus.cc:357] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 } } peers { permanent_uuid: "a665ffbad1294856aacb3a511c6ab5da" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 45411 } } peers { permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 } }
I20250814 07:58:12.594216 23557 raft_consensus.cc:383] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:58:12.594092 23559 consensus_queue.cc:260] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 } } peers { permanent_uuid: "a665ffbad1294856aacb3a511c6ab5da" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 45411 } } peers { permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 } }
I20250814 07:58:12.594506 23557 raft_consensus.cc:738] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e407e11d3dc749af97f55ebb6d29c6df, State: Initialized, Role: FOLLOWER
I20250814 07:58:12.594501 23558 consensus_queue.cc:260] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 } } peers { permanent_uuid: "a665ffbad1294856aacb3a511c6ab5da" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 45411 } } peers { permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 } }
I20250814 07:58:12.595463 23557 consensus_queue.cc:260] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 } } peers { permanent_uuid: "a665ffbad1294856aacb3a511c6ab5da" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 45411 } } peers { permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 } }
I20250814 07:58:12.598233 23559 ts_tablet_manager.cc:1428] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da: Time spent starting tablet: real 0.023s user 0.008s sys 0.014s
I20250814 07:58:12.603094 23557 ts_tablet_manager.cc:1428] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df: Time spent starting tablet: real 0.035s user 0.025s sys 0.008s
I20250814 07:58:12.604784 23538 heartbeater.cc:499] Master 127.20.120.62:40367 was elected leader, sending a full tablet report...
I20250814 07:58:12.606112 23558 ts_tablet_manager.cc:1428] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b: Time spent starting tablet: real 0.039s user 0.025s sys 0.013s
W20250814 07:58:12.628171 23539 tablet.cc:2378] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 07:58:12.700805 23272 tablet.cc:2378] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 07:58:12.790410 23406 tablet.cc:2378] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:58:12.866802 23563 raft_consensus.cc:491] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:58:12.867213 23563 raft_consensus.cc:513] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 } } peers { permanent_uuid: "a665ffbad1294856aacb3a511c6ab5da" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 45411 } } peers { permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 } }
I20250814 07:58:12.869446 23563 leader_election.cc:290] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e407e11d3dc749af97f55ebb6d29c6df (127.20.120.1:43623), fbb34b845fe14c048512f2e791b9564b (127.20.120.3:41249)
I20250814 07:58:12.879617 23226 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a70a8c8be85b44e3bf55b46f1bdbc4f4" candidate_uuid: "a665ffbad1294856aacb3a511c6ab5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e407e11d3dc749af97f55ebb6d29c6df" is_pre_election: true
I20250814 07:58:12.880498 23226 raft_consensus.cc:2466] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a665ffbad1294856aacb3a511c6ab5da in term 0.
I20250814 07:58:12.881695 23295 leader_election.cc:304] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a665ffbad1294856aacb3a511c6ab5da, e407e11d3dc749af97f55ebb6d29c6df; no voters:
I20250814 07:58:12.882261 23563 raft_consensus.cc:2802] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 07:58:12.882513 23563 raft_consensus.cc:491] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 07:58:12.882468 23493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a70a8c8be85b44e3bf55b46f1bdbc4f4" candidate_uuid: "a665ffbad1294856aacb3a511c6ab5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb34b845fe14c048512f2e791b9564b" is_pre_election: true
I20250814 07:58:12.882733 23563 raft_consensus.cc:3058] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:58:12.883106 23493 raft_consensus.cc:2466] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a665ffbad1294856aacb3a511c6ab5da in term 0.
I20250814 07:58:12.886966 23563 raft_consensus.cc:513] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 } } peers { permanent_uuid: "a665ffbad1294856aacb3a511c6ab5da" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 45411 } } peers { permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 } }
I20250814 07:58:12.888134 23563 leader_election.cc:290] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [CANDIDATE]: Term 1 election: Requested vote from peers e407e11d3dc749af97f55ebb6d29c6df (127.20.120.1:43623), fbb34b845fe14c048512f2e791b9564b (127.20.120.3:41249)
I20250814 07:58:12.888809 23226 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a70a8c8be85b44e3bf55b46f1bdbc4f4" candidate_uuid: "a665ffbad1294856aacb3a511c6ab5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e407e11d3dc749af97f55ebb6d29c6df"
I20250814 07:58:12.889063 23493 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a70a8c8be85b44e3bf55b46f1bdbc4f4" candidate_uuid: "a665ffbad1294856aacb3a511c6ab5da" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbb34b845fe14c048512f2e791b9564b"
I20250814 07:58:12.889222 23226 raft_consensus.cc:3058] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:58:12.889463 23493 raft_consensus.cc:3058] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:58:12.893185 23226 raft_consensus.cc:2466] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P e407e11d3dc749af97f55ebb6d29c6df [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a665ffbad1294856aacb3a511c6ab5da in term 1.
I20250814 07:58:12.893358 23493 raft_consensus.cc:2466] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P fbb34b845fe14c048512f2e791b9564b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a665ffbad1294856aacb3a511c6ab5da in term 1.
I20250814 07:58:12.893909 23295 leader_election.cc:304] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a665ffbad1294856aacb3a511c6ab5da, e407e11d3dc749af97f55ebb6d29c6df; no voters:
I20250814 07:58:12.894476 23563 raft_consensus.cc:2802] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:58:12.895987 23563 raft_consensus.cc:695] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [term 1 LEADER]: Becoming Leader. State: Replica: a665ffbad1294856aacb3a511c6ab5da, State: Running, Role: LEADER
I20250814 07:58:12.896828 23563 consensus_queue.cc:237] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 } } peers { permanent_uuid: "a665ffbad1294856aacb3a511c6ab5da" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 45411 } } peers { permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 } }
I20250814 07:58:12.908763 23081 catalog_manager.cc:5582] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da reported cstate change: term changed from 0 to 1, leader changed from <none> to a665ffbad1294856aacb3a511c6ab5da (127.20.120.2). New cstate: current_term: 1 leader_uuid: "a665ffbad1294856aacb3a511c6ab5da" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a665ffbad1294856aacb3a511c6ab5da" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 45411 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 } health_report { overall_health: UNKNOWN } } }
I20250814 07:58:12.969309 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:58:12.972251 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e407e11d3dc749af97f55ebb6d29c6df to finish bootstrapping
I20250814 07:58:12.983218 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a665ffbad1294856aacb3a511c6ab5da to finish bootstrapping
I20250814 07:58:12.992746 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver fbb34b845fe14c048512f2e791b9564b to finish bootstrapping
I20250814 07:58:13.360644 23563 consensus_queue.cc:1035] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [LEADER]: Connected to new peer: Peer: permanent_uuid: "e407e11d3dc749af97f55ebb6d29c6df" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 43623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250814 07:58:13.389981 23568 consensus_queue.cc:1035] T a70a8c8be85b44e3bf55b46f1bdbc4f4 P a665ffbad1294856aacb3a511c6ab5da [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbb34b845fe14c048512f2e791b9564b" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 41249 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250814 07:58:14.602988 23081 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:38402
I20250814 07:58:15.632149 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 23142
I20250814 07:58:15.658267 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 23275
I20250814 07:58:15.685290 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 23409
I20250814 07:58:15.710528 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 23050
2025-08-14T07:58:15Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11234 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250814 07:58:15.761276 20960 test_util.cc:276] Using random seed: -1564585630
I20250814 07:58:15.764860 20960 ts_itest-base.cc:115] Starting cluster with:
I20250814 07:58:15.765072 20960 ts_itest-base.cc:116] --------------
I20250814 07:58:15.765249 20960 ts_itest-base.cc:117] 3 tablet servers
I20250814 07:58:15.765478 20960 ts_itest-base.cc:118] 3 replicas per TS
I20250814 07:58:15.765650 20960 ts_itest-base.cc:119] --------------
2025-08-14T07:58:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T07:58:15Z Disabled control of system clock
I20250814 07:58:15.798602 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:40161
--webserver_interface=127.20.120.62
--webserver_port=0
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:40161 with env {}
W20250814 07:58:16.079555 23607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:16.080042 23607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:16.080416 23607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:16.109552 23607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:58:16.109804 23607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:16.110006 23607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:58:16.110220 23607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:58:16.143999 23607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:40161
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:16.145246 23607 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:16.146692 23607 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:16.157524 23613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:16.158056 23614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:17.319080 23616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:17.321367 23615 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1158 milliseconds
I20250814 07:58:17.321519 23607 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:17.322778 23607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:17.326015 23607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:17.327456 23607 hybrid_clock.cc:648] HybridClock initialized: now 1755158297327413 us; error 60 us; skew 500 ppm
I20250814 07:58:17.328263 23607 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:17.334817 23607 webserver.cc:480] Webserver started at http://127.20.120.62:45103/ using document root <none> and password file <none>
I20250814 07:58:17.335723 23607 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:17.335932 23607 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:17.336421 23607 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:58:17.340752 23607 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "8475732120104374b7ec894445ee12c6"
format_stamp: "Formatted at 2025-08-14 07:58:17 on dist-test-slave-87c1"
I20250814 07:58:17.341858 23607 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "8475732120104374b7ec894445ee12c6"
format_stamp: "Formatted at 2025-08-14 07:58:17 on dist-test-slave-87c1"
I20250814 07:58:17.348798 23607 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250814 07:58:17.354204 23623 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:17.355155 23607 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250814 07:58:17.355480 23607 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8475732120104374b7ec894445ee12c6"
format_stamp: "Formatted at 2025-08-14 07:58:17 on dist-test-slave-87c1"
I20250814 07:58:17.355799 23607 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:17.411960 23607 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:17.413374 23607 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:17.413781 23607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:17.481402 23607 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:40161
I20250814 07:58:17.481498 23674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:40161 every 8 connection(s)
I20250814 07:58:17.484014 23607 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 07:58:17.486924 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 23607
I20250814 07:58:17.487414 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 07:58:17.489866 23675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:17.514231 23675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Bootstrap starting.
I20250814 07:58:17.519699 23675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Neither blocks nor log segments found. Creating new log.
I20250814 07:58:17.521338 23675 log.cc:826] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:17.525321 23675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: No bootstrap required, opened a new log
I20250814 07:58:17.542553 23675 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:17.543215 23675 raft_consensus.cc:383] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:58:17.543430 23675 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8475732120104374b7ec894445ee12c6, State: Initialized, Role: FOLLOWER
I20250814 07:58:17.544024 23675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:17.544510 23675 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:17.544756 23675 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:17.545064 23675 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:58:17.548677 23675 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:17.549464 23675 leader_election.cc:304] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8475732120104374b7ec894445ee12c6; no voters:
I20250814 07:58:17.550997 23675 leader_election.cc:290] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:58:17.551703 23680 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:58:17.553900 23680 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 1 LEADER]: Becoming Leader. State: Replica: 8475732120104374b7ec894445ee12c6, State: Running, Role: LEADER
I20250814 07:58:17.554579 23680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:17.555475 23675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:58:17.564653 23682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8475732120104374b7ec894445ee12c6. Latest consensus state: current_term: 1 leader_uuid: "8475732120104374b7ec894445ee12c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } } }
I20250814 07:58:17.564935 23681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8475732120104374b7ec894445ee12c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } } }
I20250814 07:58:17.565328 23682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: This master's current role is: LEADER
I20250814 07:58:17.565486 23681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: This master's current role is: LEADER
I20250814 07:58:17.568542 23689 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:58:17.580163 23689 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:58:17.596534 23689 catalog_manager.cc:1349] Generated new cluster ID: 4165327691984017b938ffa38db7cd24
I20250814 07:58:17.596791 23689 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:58:17.612126 23689 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 07:58:17.613456 23689 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:58:17.627964 23689 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Generated new TSK 0
I20250814 07:58:17.628700 23689 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 07:58:17.646816 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:0
--local_ip_for_outbound_sockets=127.20.120.1
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40161
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250814 07:58:17.959789 23699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:17.960243 23699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:17.960716 23699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:17.989296 23699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:17.990144 23699 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 07:58:18.022323 23699 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:18.023497 23699 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:18.024950 23699 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:18.036672 23705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:18.037863 23706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:19.440279 23704 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 23699
W20250814 07:58:19.772686 23707 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1733 milliseconds
W20250814 07:58:19.770892 23699 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.733s user 0.584s sys 1.140s
W20250814 07:58:19.773558 23699 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.736s user 0.584s sys 1.141s
W20250814 07:58:19.773598 23708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:19.773886 23699 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:19.777328 23699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:19.779399 23699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:19.780733 23699 hybrid_clock.cc:648] HybridClock initialized: now 1755158299780704 us; error 32 us; skew 500 ppm
I20250814 07:58:19.781512 23699 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:19.787782 23699 webserver.cc:480] Webserver started at http://127.20.120.1:46029/ using document root <none> and password file <none>
I20250814 07:58:19.788628 23699 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:19.788837 23699 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:19.789352 23699 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:58:19.793478 23699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "73230c32551f4370926dd72ed159266e"
format_stamp: "Formatted at 2025-08-14 07:58:19 on dist-test-slave-87c1"
I20250814 07:58:19.794488 23699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "73230c32551f4370926dd72ed159266e"
format_stamp: "Formatted at 2025-08-14 07:58:19 on dist-test-slave-87c1"
I20250814 07:58:19.801548 23699 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20250814 07:58:19.806970 23715 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:19.807981 23699 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20250814 07:58:19.808274 23699 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "73230c32551f4370926dd72ed159266e"
format_stamp: "Formatted at 2025-08-14 07:58:19 on dist-test-slave-87c1"
I20250814 07:58:19.808571 23699 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:19.869874 23699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:19.871256 23699 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:19.871645 23699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:19.874454 23699 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:19.878402 23699 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:58:19.878579 23699 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:19.878830 23699 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:58:19.878981 23699 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:20.050407 23699 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:46459
I20250814 07:58:20.050566 23827 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:46459 every 8 connection(s)
I20250814 07:58:20.052956 23699 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 07:58:20.054045 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 23699
I20250814 07:58:20.054575 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 07:58:20.064337 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.2:0
--local_ip_for_outbound_sockets=127.20.120.2
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40161
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:58:20.078891 23828 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40161
I20250814 07:58:20.079411 23828 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:20.080655 23828 heartbeater.cc:507] Master 127.20.120.62:40161 requested a full tablet report, sending...
I20250814 07:58:20.083666 23640 ts_manager.cc:194] Registered new tserver with Master: 73230c32551f4370926dd72ed159266e (127.20.120.1:46459)
I20250814 07:58:20.086325 23640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:32869
W20250814 07:58:20.356799 23832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:20.357311 23832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:20.357918 23832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:20.388549 23832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:20.389431 23832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.2
I20250814 07:58:20.422416 23832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.2
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:20.423626 23832 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:20.425153 23832 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:20.435699 23839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:21.090180 23828 heartbeater.cc:499] Master 127.20.120.62:40161 was elected leader, sending a full tablet report...
W20250814 07:58:20.437621 23840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:21.567333 23842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:21.569262 23841 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1128 milliseconds
I20250814 07:58:21.569392 23832 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:21.570513 23832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:21.573637 23832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:21.575033 23832 hybrid_clock.cc:648] HybridClock initialized: now 1755158301574989 us; error 47 us; skew 500 ppm
I20250814 07:58:21.575726 23832 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:21.581696 23832 webserver.cc:480] Webserver started at http://127.20.120.2:46253/ using document root <none> and password file <none>
I20250814 07:58:21.582470 23832 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:21.582676 23832 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:21.583108 23832 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:58:21.587422 23832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "38b524e02a3b42ee8cfc21431efded38"
format_stamp: "Formatted at 2025-08-14 07:58:21 on dist-test-slave-87c1"
I20250814 07:58:21.588421 23832 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "38b524e02a3b42ee8cfc21431efded38"
format_stamp: "Formatted at 2025-08-14 07:58:21 on dist-test-slave-87c1"
I20250814 07:58:21.595140 23832 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250814 07:58:21.600307 23849 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:21.601197 23832 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250814 07:58:21.601457 23832 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "38b524e02a3b42ee8cfc21431efded38"
format_stamp: "Formatted at 2025-08-14 07:58:21 on dist-test-slave-87c1"
I20250814 07:58:21.601711 23832 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:21.649107 23832 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:21.650434 23832 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:21.650898 23832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:21.653410 23832 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:21.657292 23832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:58:21.657492 23832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:21.657681 23832 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:58:21.657811 23832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:21.787843 23832 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.2:46099
I20250814 07:58:21.787933 23961 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.2:46099 every 8 connection(s)
I20250814 07:58:21.790351 23832 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 07:58:21.794597 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 23832
I20250814 07:58:21.795296 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 07:58:21.801221 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.3:0
--local_ip_for_outbound_sockets=127.20.120.3
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40161
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:58:21.810089 23962 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40161
I20250814 07:58:21.810482 23962 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:21.811393 23962 heartbeater.cc:507] Master 127.20.120.62:40161 requested a full tablet report, sending...
I20250814 07:58:21.813424 23640 ts_manager.cc:194] Registered new tserver with Master: 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099)
I20250814 07:58:21.814512 23640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.2:34129
W20250814 07:58:22.085798 23966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:22.086284 23966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:22.086750 23966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:22.116967 23966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:22.117831 23966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.3
I20250814 07:58:22.149979 23966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.3
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:22.151214 23966 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:22.152781 23966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:22.163301 23972 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:22.817343 23962 heartbeater.cc:499] Master 127.20.120.62:40161 was elected leader, sending a full tablet report...
W20250814 07:58:22.164690 23973 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:23.567680 23971 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 23966
W20250814 07:58:23.781159 23971 kernel_stack_watchdog.cc:198] Thread 23966 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 07:58:23.781674 23966 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.618s user 0.000s sys 0.002s
W20250814 07:58:23.781998 23966 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.618s user 0.000s sys 0.002s
W20250814 07:58:23.782927 23975 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:23.785104 23974 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1615 milliseconds
I20250814 07:58:23.785108 23966 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:23.786579 23966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:23.790645 23966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:23.792135 23966 hybrid_clock.cc:648] HybridClock initialized: now 1755158303792095 us; error 33 us; skew 500 ppm
I20250814 07:58:23.793105 23966 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:23.800010 23966 webserver.cc:480] Webserver started at http://127.20.120.3:35023/ using document root <none> and password file <none>
I20250814 07:58:23.801203 23966 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:23.801486 23966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:23.801898 23966 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:58:23.805825 23966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "7b744475f7264e8b82e4fd5b665570dc"
format_stamp: "Formatted at 2025-08-14 07:58:23 on dist-test-slave-87c1"
I20250814 07:58:23.806835 23966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "7b744475f7264e8b82e4fd5b665570dc"
format_stamp: "Formatted at 2025-08-14 07:58:23 on dist-test-slave-87c1"
I20250814 07:58:23.812935 23966 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20250814 07:58:23.818116 23982 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:23.819016 23966 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250814 07:58:23.819332 23966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7b744475f7264e8b82e4fd5b665570dc"
format_stamp: "Formatted at 2025-08-14 07:58:23 on dist-test-slave-87c1"
I20250814 07:58:23.819615 23966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:23.866645 23966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:23.867897 23966 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:23.868274 23966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:23.870748 23966 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:23.874538 23966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:58:23.874733 23966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:23.874965 23966 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:58:23.875120 23966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:24.005735 23966 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.3:39755
I20250814 07:58:24.005856 24094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.3:39755 every 8 connection(s)
I20250814 07:58:24.008158 23966 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 07:58:24.016522 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 23966
I20250814 07:58:24.016894 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 07:58:24.030876 24095 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40161
I20250814 07:58:24.031294 24095 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:24.032270 24095 heartbeater.cc:507] Master 127.20.120.62:40161 requested a full tablet report, sending...
I20250814 07:58:24.034534 23640 ts_manager.cc:194] Registered new tserver with Master: 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755)
I20250814 07:58:24.035857 23640 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.3:55713
I20250814 07:58:24.036254 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:58:24.071422 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:58:24.071707 20960 test_util.cc:276] Using random seed: -1556275187
I20250814 07:58:24.109512 23640 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42018:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250814 07:58:24.148386 24030 tablet_service.cc:1468] Processing CreateTablet for tablet a34062205d144210a396ae4b41bd9782 (DEFAULT_TABLE table=TestTable [id=616b45b8d832414e95c194193f2a4689]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:58:24.149756 24030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34062205d144210a396ae4b41bd9782. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:24.168066 24115 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap starting.
I20250814 07:58:24.172936 24115 tablet_bootstrap.cc:654] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Neither blocks nor log segments found. Creating new log.
I20250814 07:58:24.174667 24115 log.cc:826] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:24.178809 24115 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: No bootstrap required, opened a new log
I20250814 07:58:24.179188 24115 ts_tablet_manager.cc:1397] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250814 07:58:24.195133 24115 raft_consensus.cc:357] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } }
I20250814 07:58:24.195659 24115 raft_consensus.cc:383] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:58:24.195943 24115 raft_consensus.cc:738] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b744475f7264e8b82e4fd5b665570dc, State: Initialized, Role: FOLLOWER
I20250814 07:58:24.196614 24115 consensus_queue.cc:260] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } }
I20250814 07:58:24.197103 24115 raft_consensus.cc:397] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:24.197314 24115 raft_consensus.cc:491] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:24.197551 24115 raft_consensus.cc:3058] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:58:24.201825 24115 raft_consensus.cc:513] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } }
I20250814 07:58:24.202637 24115 leader_election.cc:304] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b744475f7264e8b82e4fd5b665570dc; no voters:
I20250814 07:58:24.204234 24115 leader_election.cc:290] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:58:24.204573 24117 raft_consensus.cc:2802] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:58:24.207083 24117 raft_consensus.cc:695] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 1 LEADER]: Becoming Leader. State: Replica: 7b744475f7264e8b82e4fd5b665570dc, State: Running, Role: LEADER
I20250814 07:58:24.207849 24095 heartbeater.cc:499] Master 127.20.120.62:40161 was elected leader, sending a full tablet report...
I20250814 07:58:24.207947 24117 consensus_queue.cc:237] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } }
I20250814 07:58:24.209074 24115 ts_tablet_manager.cc:1428] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Time spent starting tablet: real 0.030s user 0.027s sys 0.004s
I20250814 07:58:24.219067 23640 catalog_manager.cc:5582] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc reported cstate change: term changed from 0 to 1, leader changed from <none> to 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3). New cstate: current_term: 1 leader_uuid: "7b744475f7264e8b82e4fd5b665570dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } health_report { overall_health: HEALTHY } } }
I20250814 07:58:24.366494 20960 test_util.cc:276] Using random seed: -1555980418
I20250814 07:58:24.386265 23639 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42020:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250814 07:58:24.413599 23763 tablet_service.cc:1468] Processing CreateTablet for tablet 967c771199494673af94bd0b45841b7d (DEFAULT_TABLE table=TestTable1 [id=143cfcab7a9b476b817a9bb28dd4f3fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:58:24.415050 23763 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 967c771199494673af94bd0b45841b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:24.432559 24136 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Bootstrap starting.
I20250814 07:58:24.438109 24136 tablet_bootstrap.cc:654] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Neither blocks nor log segments found. Creating new log.
I20250814 07:58:24.439720 24136 log.cc:826] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:24.443709 24136 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: No bootstrap required, opened a new log
I20250814 07:58:24.444089 24136 ts_tablet_manager.cc:1397] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.004s
I20250814 07:58:24.460602 24136 raft_consensus.cc:357] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } }
I20250814 07:58:24.461163 24136 raft_consensus.cc:383] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:58:24.461373 24136 raft_consensus.cc:738] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73230c32551f4370926dd72ed159266e, State: Initialized, Role: FOLLOWER
I20250814 07:58:24.461979 24136 consensus_queue.cc:260] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } }
I20250814 07:58:24.462495 24136 raft_consensus.cc:397] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:24.462759 24136 raft_consensus.cc:491] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:24.463124 24136 raft_consensus.cc:3058] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:58:24.467123 24136 raft_consensus.cc:513] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } }
I20250814 07:58:24.467748 24136 leader_election.cc:304] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73230c32551f4370926dd72ed159266e; no voters:
I20250814 07:58:24.469558 24136 leader_election.cc:290] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:58:24.469960 24138 raft_consensus.cc:2802] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:58:24.472736 24138 raft_consensus.cc:695] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 1 LEADER]: Becoming Leader. State: Replica: 73230c32551f4370926dd72ed159266e, State: Running, Role: LEADER
I20250814 07:58:24.473574 24138 consensus_queue.cc:237] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } }
I20250814 07:58:24.474099 24136 ts_tablet_manager.cc:1428] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Time spent starting tablet: real 0.030s user 0.022s sys 0.009s
I20250814 07:58:24.483970 23639 catalog_manager.cc:5582] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e reported cstate change: term changed from 0 to 1, leader changed from <none> to 73230c32551f4370926dd72ed159266e (127.20.120.1). New cstate: current_term: 1 leader_uuid: "73230c32551f4370926dd72ed159266e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } health_report { overall_health: HEALTHY } } }
I20250814 07:58:24.685923 20960 test_util.cc:276] Using random seed: -1555660990
I20250814 07:58:24.705621 23639 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42032:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250814 07:58:24.732414 23897 tablet_service.cc:1468] Processing CreateTablet for tablet 056a8fdaa76046ad8db0b7f717d239f8 (DEFAULT_TABLE table=TestTable2 [id=c510acfde0144ac08f7d263d93711de5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:58:24.733887 23897 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 056a8fdaa76046ad8db0b7f717d239f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:24.752197 24156 tablet_bootstrap.cc:492] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap starting.
I20250814 07:58:24.757261 24156 tablet_bootstrap.cc:654] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Neither blocks nor log segments found. Creating new log.
I20250814 07:58:24.758778 24156 log.cc:826] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:24.762701 24156 tablet_bootstrap.cc:492] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: No bootstrap required, opened a new log
I20250814 07:58:24.763022 24156 ts_tablet_manager.cc:1397] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20250814 07:58:24.779121 24156 raft_consensus.cc:357] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:24.779601 24156 raft_consensus.cc:383] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:58:24.779776 24156 raft_consensus.cc:738] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b524e02a3b42ee8cfc21431efded38, State: Initialized, Role: FOLLOWER
I20250814 07:58:24.780347 24156 consensus_queue.cc:260] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:24.780773 24156 raft_consensus.cc:397] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:24.781013 24156 raft_consensus.cc:491] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:24.781275 24156 raft_consensus.cc:3058] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:58:24.784839 24156 raft_consensus.cc:513] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:24.785497 24156 leader_election.cc:304] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b524e02a3b42ee8cfc21431efded38; no voters:
I20250814 07:58:24.787303 24156 leader_election.cc:290] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:58:24.787850 24158 raft_consensus.cc:2802] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:58:24.793327 24156 ts_tablet_manager.cc:1428] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Time spent starting tablet: real 0.030s user 0.032s sys 0.000s
I20250814 07:58:24.793589 24158 raft_consensus.cc:695] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 1 LEADER]: Becoming Leader. State: Replica: 38b524e02a3b42ee8cfc21431efded38, State: Running, Role: LEADER
I20250814 07:58:24.794399 24158 consensus_queue.cc:237] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
W20250814 07:58:24.798794 23963 tablet.cc:2378] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:58:24.806859 23639 catalog_manager.cc:5582] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2). New cstate: current_term: 1 leader_uuid: "38b524e02a3b42ee8cfc21431efded38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } health_report { overall_health: HEALTHY } } }
I20250814 07:58:24.989750 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 23607
W20250814 07:58:25.232784 24095 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:40161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:40161: connect: Connection refused (error 111)
W20250814 07:58:25.499148 23828 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:40161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:40161: connect: Connection refused (error 111)
W20250814 07:58:25.836210 23962 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:40161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:40161: connect: Connection refused (error 111)
I20250814 07:58:29.692832 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 23699
I20250814 07:58:29.715526 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 23832
I20250814 07:58:29.740639 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 23966
I20250814 07:58:29.766203 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:40161
--webserver_interface=127.20.120.62
--webserver_port=45103
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:40161 with env {}
W20250814 07:58:30.072006 24234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:30.072579 24234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:30.073056 24234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:30.104256 24234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:58:30.104557 24234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:30.104810 24234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:58:30.105060 24234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:58:30.138926 24234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:40161
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=45103
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:30.140153 24234 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:30.141683 24234 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:30.151063 24240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:30.152805 24241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:31.314100 24243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:31.316092 24242 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1159 milliseconds
I20250814 07:58:31.316221 24234 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:31.317454 24234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:31.320456 24234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:31.321882 24234 hybrid_clock.cc:648] HybridClock initialized: now 1755158311321835 us; error 48 us; skew 500 ppm
I20250814 07:58:31.322649 24234 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:31.328845 24234 webserver.cc:480] Webserver started at http://127.20.120.62:45103/ using document root <none> and password file <none>
I20250814 07:58:31.329775 24234 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:31.329984 24234 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:31.337491 24234 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250814 07:58:31.341713 24251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:31.342633 24234 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250814 07:58:31.342941 24234 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8475732120104374b7ec894445ee12c6"
format_stamp: "Formatted at 2025-08-14 07:58:17 on dist-test-slave-87c1"
I20250814 07:58:31.344720 24234 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:31.400454 24234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:31.401854 24234 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:31.402287 24234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:31.468401 24234 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:40161
I20250814 07:58:31.468456 24302 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:40161 every 8 connection(s)
I20250814 07:58:31.471118 24234 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 07:58:31.479683 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 24234
I20250814 07:58:31.480592 24303 sys_catalog.cc:263] Verifying existing consensus state
I20250814 07:58:31.481325 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:46459
--local_ip_for_outbound_sockets=127.20.120.1
--tserver_master_addrs=127.20.120.62:40161
--webserver_port=46029
--webserver_interface=127.20.120.1
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:58:31.485041 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Bootstrap starting.
I20250814 07:58:31.496563 24303 log.cc:826] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:31.555820 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:31.556651 24303 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Bootstrap complete.
I20250814 07:58:31.577973 24303 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:31.580260 24303 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8475732120104374b7ec894445ee12c6, State: Initialized, Role: FOLLOWER
I20250814 07:58:31.580920 24303 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:31.581497 24303 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:31.581759 24303 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:31.582056 24303 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 2 FOLLOWER]: Advancing to term 3
I20250814 07:58:31.588827 24303 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:31.589437 24303 leader_election.cc:304] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8475732120104374b7ec894445ee12c6; no voters:
I20250814 07:58:31.591158 24303 leader_election.cc:290] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250814 07:58:31.591609 24307 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 3 FOLLOWER]: Leader election won for term 3
I20250814 07:58:31.594566 24307 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 3 LEADER]: Becoming Leader. State: Replica: 8475732120104374b7ec894445ee12c6, State: Running, Role: LEADER
I20250814 07:58:31.595487 24303 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:58:31.595459 24307 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:31.605856 24309 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8475732120104374b7ec894445ee12c6. Latest consensus state: current_term: 3 leader_uuid: "8475732120104374b7ec894445ee12c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } } }
I20250814 07:58:31.606696 24309 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: This master's current role is: LEADER
I20250814 07:58:31.615617 24308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "8475732120104374b7ec894445ee12c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } } }
I20250814 07:58:31.616227 24308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: This master's current role is: LEADER
I20250814 07:58:31.619153 24313 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:58:31.630852 24313 catalog_manager.cc:671] Loaded metadata for table TestTable [id=93efa396640e4ce59aec591d5afedf53]
I20250814 07:58:31.632467 24313 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=b565d9c2cccc451f9a8999179837e866]
I20250814 07:58:31.634155 24313 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=c510acfde0144ac08f7d263d93711de5]
I20250814 07:58:31.642431 24313 tablet_loader.cc:96] loaded metadata for tablet 056a8fdaa76046ad8db0b7f717d239f8 (table TestTable2 [id=c510acfde0144ac08f7d263d93711de5])
I20250814 07:58:31.643682 24313 tablet_loader.cc:96] loaded metadata for tablet 967c771199494673af94bd0b45841b7d (table TestTable1 [id=b565d9c2cccc451f9a8999179837e866])
I20250814 07:58:31.644850 24313 tablet_loader.cc:96] loaded metadata for tablet a34062205d144210a396ae4b41bd9782 (table TestTable [id=93efa396640e4ce59aec591d5afedf53])
I20250814 07:58:31.646277 24313 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:58:31.651289 24313 catalog_manager.cc:1261] Loaded cluster ID: 4165327691984017b938ffa38db7cd24
I20250814 07:58:31.651585 24313 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:58:31.660548 24313 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:58:31.665827 24313 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Loaded TSK: 0
I20250814 07:58:31.667312 24313 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250814 07:58:31.818614 24305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:31.819077 24305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:31.819545 24305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:31.848644 24305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:31.849438 24305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 07:58:31.882172 24305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:46459
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=46029
--tserver_master_addrs=127.20.120.62:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:31.883457 24305 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:31.884968 24305 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:31.896863 24330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:33.300006 24329 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 24305
W20250814 07:58:33.315842 24305 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.418s user 0.468s sys 0.901s
W20250814 07:58:33.316314 24305 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.418s user 0.468s sys 0.901s
W20250814 07:58:31.898222 24331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:33.317551 24332 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1418 milliseconds
W20250814 07:58:33.318568 24333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:33.318511 24305 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:33.321543 24305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:33.323628 24305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:33.324972 24305 hybrid_clock.cc:648] HybridClock initialized: now 1755158313324940 us; error 31 us; skew 500 ppm
I20250814 07:58:33.325750 24305 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:33.332145 24305 webserver.cc:480] Webserver started at http://127.20.120.1:46029/ using document root <none> and password file <none>
I20250814 07:58:33.332995 24305 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:33.333298 24305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:33.340977 24305 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20250814 07:58:33.347307 24340 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:33.348315 24305 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250814 07:58:33.348609 24305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "73230c32551f4370926dd72ed159266e"
format_stamp: "Formatted at 2025-08-14 07:58:19 on dist-test-slave-87c1"
I20250814 07:58:33.350489 24305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:33.398512 24305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:33.399899 24305 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:33.400339 24305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:33.403402 24305 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:33.409420 24347 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 07:58:33.416694 24305 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 07:58:33.416906 24305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.000s sys 0.002s
I20250814 07:58:33.417229 24305 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 07:58:33.421456 24305 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 07:58:33.421697 24305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20250814 07:58:33.422013 24347 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Bootstrap starting.
I20250814 07:58:33.502883 24347 log.cc:826] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:33.634532 24305 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:46459
I20250814 07:58:33.634600 24347 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:33.634712 24454 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:46459 every 8 connection(s)
I20250814 07:58:33.635847 24347 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Bootstrap complete.
I20250814 07:58:33.637436 24305 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 07:58:33.637634 24347 ts_tablet_manager.cc:1397] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Time spent bootstrapping tablet: real 0.216s user 0.165s sys 0.036s
I20250814 07:58:33.642385 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 24305
I20250814 07:58:33.644178 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.2:46099
--local_ip_for_outbound_sockets=127.20.120.2
--tserver_master_addrs=127.20.120.62:40161
--webserver_port=46253
--webserver_interface=127.20.120.2
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:58:33.668450 24347 raft_consensus.cc:357] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } }
I20250814 07:58:33.671554 24347 raft_consensus.cc:738] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73230c32551f4370926dd72ed159266e, State: Initialized, Role: FOLLOWER
I20250814 07:58:33.672673 24347 consensus_queue.cc:260] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } }
I20250814 07:58:33.673496 24347 raft_consensus.cc:397] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:33.673911 24347 raft_consensus.cc:491] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:33.674369 24347 raft_consensus.cc:3058] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 1 FOLLOWER]: Advancing to term 2
I20250814 07:58:33.683184 24347 raft_consensus.cc:513] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } }
I20250814 07:58:33.684265 24347 leader_election.cc:304] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73230c32551f4370926dd72ed159266e; no voters:
I20250814 07:58:33.688786 24455 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40161
I20250814 07:58:33.689495 24455 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:33.691130 24455 heartbeater.cc:507] Master 127.20.120.62:40161 requested a full tablet report, sending...
I20250814 07:58:33.697603 24347 leader_election.cc:290] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 2 election: Requested vote from peers
I20250814 07:58:33.697625 24268 ts_manager.cc:194] Registered new tserver with Master: 73230c32551f4370926dd72ed159266e (127.20.120.1:46459)
I20250814 07:58:33.698035 24460 raft_consensus.cc:2802] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Leader election won for term 2
I20250814 07:58:33.707041 24460 raft_consensus.cc:695] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 LEADER]: Becoming Leader. State: Replica: 73230c32551f4370926dd72ed159266e, State: Running, Role: LEADER
I20250814 07:58:33.708040 24460 consensus_queue.cc:237] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } }
I20250814 07:58:33.708473 24268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:42853
I20250814 07:58:33.717489 24455 heartbeater.cc:499] Master 127.20.120.62:40161 was elected leader, sending a full tablet report...
I20250814 07:58:33.719038 24347 ts_tablet_manager.cc:1428] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Time spent starting tablet: real 0.081s user 0.052s sys 0.016s
I20250814 07:58:33.728283 24268 catalog_manager.cc:5582] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e reported cstate change: term changed from 0 to 2, leader changed from <none> to 73230c32551f4370926dd72ed159266e (127.20.120.1), VOTER 73230c32551f4370926dd72ed159266e (127.20.120.1) added. New cstate: current_term: 2 leader_uuid: "73230c32551f4370926dd72ed159266e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } health_report { overall_health: HEALTHY } } }
W20250814 07:58:33.760974 24268 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 967c771199494673af94bd0b45841b7d with cas_config_opid_index -1: no extra replica candidate found for tablet 967c771199494673af94bd0b45841b7d (table TestTable1 [id=b565d9c2cccc451f9a8999179837e866]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (1) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250814 07:58:34.015444 24459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:34.015973 24459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:34.016492 24459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:34.048699 24459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:34.049638 24459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.2
I20250814 07:58:34.086078 24459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.2:46099
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.120.2
--webserver_port=46253
--tserver_master_addrs=127.20.120.62:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.2
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:34.087445 24459 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:34.089128 24459 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:34.103406 24474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:34.110384 24475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:34.112078 24477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:34.112653 24459 server_base.cc:1047] running on GCE node
I20250814 07:58:35.276539 24459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:35.278856 24459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:35.280197 24459 hybrid_clock.cc:648] HybridClock initialized: now 1755158315280179 us; error 50 us; skew 500 ppm
I20250814 07:58:35.280934 24459 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:35.286988 24459 webserver.cc:480] Webserver started at http://127.20.120.2:46253/ using document root <none> and password file <none>
I20250814 07:58:35.287883 24459 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:35.288125 24459 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:35.295979 24459 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20250814 07:58:35.300870 24484 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:35.301976 24459 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250814 07:58:35.302318 24459 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "38b524e02a3b42ee8cfc21431efded38"
format_stamp: "Formatted at 2025-08-14 07:58:21 on dist-test-slave-87c1"
I20250814 07:58:35.304381 24459 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:35.368217 24459 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:35.369653 24459 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:35.370059 24459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:35.372516 24459 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:35.378377 24491 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 07:58:35.385602 24459 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 07:58:35.385891 24459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250814 07:58:35.386171 24459 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 07:58:35.390659 24459 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 07:58:35.390842 24459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250814 07:58:35.391157 24491 tablet_bootstrap.cc:492] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap starting.
I20250814 07:58:35.442726 24491 log.cc:826] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:35.530958 24491 tablet_bootstrap.cc:492] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:35.531661 24491 tablet_bootstrap.cc:492] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap complete.
I20250814 07:58:35.532944 24491 ts_tablet_manager.cc:1397] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Time spent bootstrapping tablet: real 0.142s user 0.105s sys 0.032s
I20250814 07:58:35.548528 24491 raft_consensus.cc:357] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:35.551259 24491 raft_consensus.cc:738] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b524e02a3b42ee8cfc21431efded38, State: Initialized, Role: FOLLOWER
I20250814 07:58:35.552210 24491 consensus_queue.cc:260] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:35.552969 24491 raft_consensus.cc:397] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:35.553344 24491 raft_consensus.cc:491] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:35.553683 24491 raft_consensus.cc:3058] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 1 FOLLOWER]: Advancing to term 2
I20250814 07:58:35.559396 24491 raft_consensus.cc:513] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:35.560014 24491 leader_election.cc:304] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b524e02a3b42ee8cfc21431efded38; no voters:
I20250814 07:58:35.562495 24491 leader_election.cc:290] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250814 07:58:35.563441 24596 raft_consensus.cc:2802] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 07:58:35.566851 24596 raft_consensus.cc:695] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 2 LEADER]: Becoming Leader. State: Replica: 38b524e02a3b42ee8cfc21431efded38, State: Running, Role: LEADER
I20250814 07:58:35.567760 24491 ts_tablet_manager.cc:1428] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Time spent starting tablet: real 0.035s user 0.032s sys 0.001s
I20250814 07:58:35.568044 24596 consensus_queue.cc:237] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:35.568987 24459 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.2:46099
I20250814 07:58:35.569200 24601 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.2:46099 every 8 connection(s)
I20250814 07:58:35.572348 24459 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 07:58:35.578521 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 24459
I20250814 07:58:35.580127 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.3:39755
--local_ip_for_outbound_sockets=127.20.120.3
--tserver_master_addrs=127.20.120.62:40161
--webserver_port=35023
--webserver_interface=127.20.120.3
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:58:35.596029 24604 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40161
I20250814 07:58:35.596537 24604 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:35.597873 24604 heartbeater.cc:507] Master 127.20.120.62:40161 requested a full tablet report, sending...
I20250814 07:58:35.602038 24268 ts_manager.cc:194] Registered new tserver with Master: 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099)
I20250814 07:58:35.602983 24268 catalog_manager.cc:5582] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "38b524e02a3b42ee8cfc21431efded38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } health_report { overall_health: HEALTHY } } }
I20250814 07:58:35.613469 24268 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.2:34911
I20250814 07:58:35.616734 24604 heartbeater.cc:499] Master 127.20.120.62:40161 was elected leader, sending a full tablet report...
W20250814 07:58:35.872483 24608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:35.872890 24608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:35.873414 24608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:35.902765 24608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:35.903498 24608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.3
I20250814 07:58:35.936002 24608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.3:39755
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.120.3
--webserver_port=35023
--tserver_master_addrs=127.20.120.62:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.3
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:35.937229 24608 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:35.938642 24608 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:35.949982 24618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:36.096451 24410 consensus_queue.cc:237] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } }
I20250814 07:58:36.115734 24628 raft_consensus.cc:2953] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } } }
I20250814 07:58:36.143911 24254 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 967c771199494673af94bd0b45841b7d with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 8)
I20250814 07:58:36.151110 24268 catalog_manager.cc:5582] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e reported cstate change: config changed from index -1 to 9, NON_VOTER 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) added. New cstate: current_term: 2 leader_uuid: "73230c32551f4370926dd72ed159266e" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250814 07:58:36.170120 24343 consensus_peers.cc:489] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e -> Peer 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099): Couldn't send request to peer 38b524e02a3b42ee8cfc21431efded38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 967c771199494673af94bd0b45841b7d. This is attempt 1: this message will repeat every 5th retry.
I20250814 07:58:36.610008 24634 ts_tablet_manager.cc:927] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Initiating tablet copy from peer 73230c32551f4370926dd72ed159266e (127.20.120.1:46459)
I20250814 07:58:36.615979 24634 tablet_copy_client.cc:323] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: tablet copy: Beginning tablet copy session from remote peer at address 127.20.120.1:46459
I20250814 07:58:36.667939 24430 tablet_copy_service.cc:140] P 73230c32551f4370926dd72ed159266e: Received BeginTabletCopySession request for tablet 967c771199494673af94bd0b45841b7d from peer 38b524e02a3b42ee8cfc21431efded38 ({username='slave'} at 127.20.120.2:46715)
I20250814 07:58:36.668800 24430 tablet_copy_service.cc:161] P 73230c32551f4370926dd72ed159266e: Beginning new tablet copy session on tablet 967c771199494673af94bd0b45841b7d from peer 38b524e02a3b42ee8cfc21431efded38 at {username='slave'} at 127.20.120.2:46715: session id = 38b524e02a3b42ee8cfc21431efded38-967c771199494673af94bd0b45841b7d
I20250814 07:58:36.680830 24430 tablet_copy_source_session.cc:215] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Tablet Copy: opened 0 blocks and 1 log segments
I20250814 07:58:36.686467 24634 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 967c771199494673af94bd0b45841b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:36.706580 24634 tablet_copy_client.cc:806] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: tablet copy: Starting download of 0 data blocks...
I20250814 07:58:36.707330 24634 tablet_copy_client.cc:670] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: tablet copy: Starting download of 1 WAL segments...
I20250814 07:58:36.712749 24634 tablet_copy_client.cc:538] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250814 07:58:36.723527 24634 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Bootstrap starting.
I20250814 07:58:36.891656 24634 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:36.892828 24634 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Bootstrap complete.
I20250814 07:58:36.893685 24634 ts_tablet_manager.cc:1397] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Time spent bootstrapping tablet: real 0.170s user 0.157s sys 0.013s
I20250814 07:58:36.896803 24634 raft_consensus.cc:357] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } }
I20250814 07:58:36.897701 24634 raft_consensus.cc:738] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 38b524e02a3b42ee8cfc21431efded38, State: Initialized, Role: LEARNER
I20250814 07:58:36.898509 24634 consensus_queue.cc:260] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } }
I20250814 07:58:36.908283 24634 ts_tablet_manager.cc:1428] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Time spent starting tablet: real 0.014s user 0.008s sys 0.004s
I20250814 07:58:36.923624 24430 tablet_copy_service.cc:342] P 73230c32551f4370926dd72ed159266e: Request end of tablet copy session 38b524e02a3b42ee8cfc21431efded38-967c771199494673af94bd0b45841b7d received from {username='slave'} at 127.20.120.2:46715
I20250814 07:58:36.924198 24430 tablet_copy_service.cc:434] P 73230c32551f4370926dd72ed159266e: ending tablet copy session 38b524e02a3b42ee8cfc21431efded38-967c771199494673af94bd0b45841b7d on tablet 967c771199494673af94bd0b45841b7d with peer 38b524e02a3b42ee8cfc21431efded38
I20250814 07:58:37.184484 24554 raft_consensus.cc:1215] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20250814 07:58:37.354467 24617 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 24608
W20250814 07:58:37.679096 24608 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.729s user 0.640s sys 1.015s
W20250814 07:58:35.950935 24619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:37.680325 24608 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.730s user 0.640s sys 1.015s
W20250814 07:58:37.681227 24620 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1730 milliseconds
I20250814 07:58:37.681658 24608 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250814 07:58:37.681743 24621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:37.686234 24608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:37.688434 24608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:37.689849 24608 hybrid_clock.cc:648] HybridClock initialized: now 1755158317689824 us; error 47 us; skew 500 ppm
I20250814 07:58:37.690652 24608 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:37.695060 24645 raft_consensus.cc:1062] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: attempting to promote NON_VOTER 38b524e02a3b42ee8cfc21431efded38 to VOTER
I20250814 07:58:37.696789 24608 webserver.cc:480] Webserver started at http://127.20.120.3:35023/ using document root <none> and password file <none>
I20250814 07:58:37.696595 24645 consensus_queue.cc:237] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:37.697774 24608 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:37.698022 24608 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:37.700655 24554 raft_consensus.cc:1273] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 LEARNER]: Refusing update from remote peer 73230c32551f4370926dd72ed159266e: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250814 07:58:37.701782 24643 consensus_queue.cc:1035] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250814 07:58:37.707058 24608 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20250814 07:58:37.707978 24645 raft_consensus.cc:2953] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:37.709537 24554 raft_consensus.cc:2953] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:37.712538 24650 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:37.713544 24608 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250814 07:58:37.713847 24608 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7b744475f7264e8b82e4fd5b665570dc"
format_stamp: "Formatted at 2025-08-14 07:58:23 on dist-test-slave-87c1"
I20250814 07:58:37.715987 24608 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:37.719131 24267 catalog_manager.cc:5582] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e reported cstate change: config changed from index 9 to 10, 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "73230c32551f4370926dd72ed159266e" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250814 07:58:37.769928 24608 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:37.771234 24608 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:37.771647 24608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:37.774559 24608 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:37.779878 24662 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 07:58:37.786609 24608 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 07:58:37.786844 24608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.002s sys 0.000s
I20250814 07:58:37.787093 24608 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 07:58:37.791569 24608 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 07:58:37.791750 24608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250814 07:58:37.792043 24662 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap starting.
I20250814 07:58:37.842412 24662 log.cc:826] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:37.954126 24608 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.3:39755
I20250814 07:58:37.954232 24769 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.3:39755 every 8 connection(s)
I20250814 07:58:37.956465 24608 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 07:58:37.961980 24662 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:37.964149 24662 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap complete.
I20250814 07:58:37.965083 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 24608
I20250814 07:58:37.966298 24662 ts_tablet_manager.cc:1397] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Time spent bootstrapping tablet: real 0.175s user 0.144s sys 0.024s
I20250814 07:58:37.987771 24662 raft_consensus.cc:357] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } }
I20250814 07:58:37.990459 24662 raft_consensus.cc:738] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b744475f7264e8b82e4fd5b665570dc, State: Initialized, Role: FOLLOWER
I20250814 07:58:37.991269 24662 consensus_queue.cc:260] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } }
I20250814 07:58:37.991858 24662 raft_consensus.cc:397] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:37.992168 24662 raft_consensus.cc:491] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:37.992620 24662 raft_consensus.cc:3058] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 1 FOLLOWER]: Advancing to term 2
I20250814 07:58:38.001979 24662 raft_consensus.cc:513] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } }
I20250814 07:58:38.002902 24662 leader_election.cc:304] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b744475f7264e8b82e4fd5b665570dc; no voters:
I20250814 07:58:38.009119 24770 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40161
I20250814 07:58:38.009588 24770 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:38.010036 24662 leader_election.cc:290] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [CANDIDATE]: Term 2 election: Requested vote from peers
I20250814 07:58:38.010448 24776 raft_consensus.cc:2802] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 FOLLOWER]: Leader election won for term 2
I20250814 07:58:38.010896 24770 heartbeater.cc:507] Master 127.20.120.62:40161 requested a full tablet report, sending...
I20250814 07:58:38.013969 24776 raft_consensus.cc:695] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 LEADER]: Becoming Leader. State: Replica: 7b744475f7264e8b82e4fd5b665570dc, State: Running, Role: LEADER
I20250814 07:58:38.014377 24662 ts_tablet_manager.cc:1428] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Time spent starting tablet: real 0.047s user 0.036s sys 0.008s
I20250814 07:58:38.014720 24776 consensus_queue.cc:237] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } }
I20250814 07:58:38.025283 24267 ts_manager.cc:194] Registered new tserver with Master: 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755)
I20250814 07:58:38.026221 24267 catalog_manager.cc:5582] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc reported cstate change: term changed from 0 to 2, leader changed from <none> to 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3), VOTER 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) added. New cstate: current_term: 2 leader_uuid: "7b744475f7264e8b82e4fd5b665570dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } health_report { overall_health: HEALTHY } } }
I20250814 07:58:38.038103 24267 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.3:58805
I20250814 07:58:38.039031 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:58:38.041987 24770 heartbeater.cc:499] Master 127.20.120.62:40161 was elected leader, sending a full tablet report...
I20250814 07:58:38.043812 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250814 07:58:38.047832 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250814 07:58:38.051357 24725 consensus_queue.cc:237] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } }
I20250814 07:58:38.054831 24778 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index -1 to 8, NON_VOTER 73230c32551f4370926dd72ed159266e (127.20.120.1) added. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } }
I20250814 07:58:38.063241 24254 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a34062205d144210a396ae4b41bd9782 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250814 07:58:38.064904 24658 consensus_peers.cc:489] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc -> Peer 73230c32551f4370926dd72ed159266e (127.20.120.1:46459): Couldn't send request to peer 73230c32551f4370926dd72ed159266e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a34062205d144210a396ae4b41bd9782. This is attempt 1: this message will repeat every 5th retry.
I20250814 07:58:38.065759 24268 catalog_manager.cc:5582] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc reported cstate change: config changed from index -1 to 8, NON_VOTER 73230c32551f4370926dd72ed159266e (127.20.120.1) added. New cstate: current_term: 2 leader_uuid: "7b744475f7264e8b82e4fd5b665570dc" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250814 07:58:38.070298 24410 consensus_queue.cc:237] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: NON_VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: true } }
I20250814 07:58:38.075134 24725 consensus_queue.cc:237] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } }
I20250814 07:58:38.076651 24554 raft_consensus.cc:1273] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Refusing update from remote peer 73230c32551f4370926dd72ed159266e: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250814 07:58:38.077828 24645 consensus_queue.cc:1035] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250814 07:58:38.079588 24778 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, NON_VOTER 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } } }
I20250814 07:58:38.086774 24645 raft_consensus.cc:2953] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: NON_VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: true } } }
W20250814 07:58:38.089619 24343 consensus_peers.cc:489] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e -> Peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755): Couldn't send request to peer 7b744475f7264e8b82e4fd5b665570dc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 967c771199494673af94bd0b45841b7d. This is attempt 1: this message will repeat every 5th retry.
I20250814 07:58:38.090153 24254 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a34062205d144210a396ae4b41bd9782 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250814 07:58:38.091089 24658 consensus_peers.cc:489] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc -> Peer 73230c32551f4370926dd72ed159266e (127.20.120.1:46459): Couldn't send request to peer 73230c32551f4370926dd72ed159266e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a34062205d144210a396ae4b41bd9782. This is attempt 1: this message will repeat every 5th retry.
I20250814 07:58:38.090667 24554 raft_consensus.cc:2953] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: NON_VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: true } } }
I20250814 07:58:38.093720 24268 catalog_manager.cc:5582] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc reported cstate change: config changed from index 8 to 9, NON_VOTER 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) added. New cstate: current_term: 2 leader_uuid: "7b744475f7264e8b82e4fd5b665570dc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250814 07:58:38.097474 24254 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 967c771199494673af94bd0b45841b7d with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250814 07:58:38.100298 24267 catalog_manager.cc:5582] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e reported cstate change: config changed from index 10 to 11, NON_VOTER 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) added. New cstate: current_term: 2 leader_uuid: "73230c32551f4370926dd72ed159266e" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: NON_VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250814 07:58:38.101490 24658 consensus_peers.cc:489] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc -> Peer 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099): Couldn't send request to peer 38b524e02a3b42ee8cfc21431efded38. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a34062205d144210a396ae4b41bd9782. This is attempt 1: this message will repeat every 5th retry.
I20250814 07:58:38.423659 24254 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 967c771199494673af94bd0b45841b7d with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250814 07:58:38.592486 24786 ts_tablet_manager.cc:927] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Initiating tablet copy from peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755)
I20250814 07:58:38.593748 24786 tablet_copy_client.cc:323] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: tablet copy: Beginning tablet copy session from remote peer at address 127.20.120.3:39755
I20250814 07:58:38.603400 24745 tablet_copy_service.cc:140] P 7b744475f7264e8b82e4fd5b665570dc: Received BeginTabletCopySession request for tablet a34062205d144210a396ae4b41bd9782 from peer 38b524e02a3b42ee8cfc21431efded38 ({username='slave'} at 127.20.120.2:35645)
I20250814 07:58:38.603978 24745 tablet_copy_service.cc:161] P 7b744475f7264e8b82e4fd5b665570dc: Beginning new tablet copy session on tablet a34062205d144210a396ae4b41bd9782 from peer 38b524e02a3b42ee8cfc21431efded38 at {username='slave'} at 127.20.120.2:35645: session id = 38b524e02a3b42ee8cfc21431efded38-a34062205d144210a396ae4b41bd9782
I20250814 07:58:38.609467 24745 tablet_copy_source_session.cc:215] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Tablet Copy: opened 0 blocks and 1 log segments
I20250814 07:58:38.612608 24786 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34062205d144210a396ae4b41bd9782. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:38.614254 24791 ts_tablet_manager.cc:927] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Initiating tablet copy from peer 73230c32551f4370926dd72ed159266e (127.20.120.1:46459)
I20250814 07:58:38.616535 24791 tablet_copy_client.cc:323] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: tablet copy: Beginning tablet copy session from remote peer at address 127.20.120.1:46459
I20250814 07:58:38.617812 24430 tablet_copy_service.cc:140] P 73230c32551f4370926dd72ed159266e: Received BeginTabletCopySession request for tablet 967c771199494673af94bd0b45841b7d from peer 7b744475f7264e8b82e4fd5b665570dc ({username='slave'} at 127.20.120.3:48835)
I20250814 07:58:38.618184 24430 tablet_copy_service.cc:161] P 73230c32551f4370926dd72ed159266e: Beginning new tablet copy session on tablet 967c771199494673af94bd0b45841b7d from peer 7b744475f7264e8b82e4fd5b665570dc at {username='slave'} at 127.20.120.3:48835: session id = 7b744475f7264e8b82e4fd5b665570dc-967c771199494673af94bd0b45841b7d
I20250814 07:58:38.622160 24430 tablet_copy_source_session.cc:215] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Tablet Copy: opened 0 blocks and 1 log segments
I20250814 07:58:38.622718 24786 tablet_copy_client.cc:806] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: tablet copy: Starting download of 0 data blocks...
I20250814 07:58:38.623205 24786 tablet_copy_client.cc:670] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: tablet copy: Starting download of 1 WAL segments...
I20250814 07:58:38.624907 24791 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 967c771199494673af94bd0b45841b7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:38.626763 24786 tablet_copy_client.cc:538] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250814 07:58:38.631424 24786 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap starting.
I20250814 07:58:38.640142 24791 tablet_copy_client.cc:806] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: tablet copy: Starting download of 0 data blocks...
I20250814 07:58:38.640630 24791 tablet_copy_client.cc:670] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: tablet copy: Starting download of 1 WAL segments...
I20250814 07:58:38.644814 24791 tablet_copy_client.cc:538] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250814 07:58:38.652285 24791 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap starting.
I20250814 07:58:38.695257 24795 ts_tablet_manager.cc:927] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Initiating tablet copy from peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755)
I20250814 07:58:38.696835 24795 tablet_copy_client.cc:323] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: tablet copy: Beginning tablet copy session from remote peer at address 127.20.120.3:39755
I20250814 07:58:38.698130 24745 tablet_copy_service.cc:140] P 7b744475f7264e8b82e4fd5b665570dc: Received BeginTabletCopySession request for tablet a34062205d144210a396ae4b41bd9782 from peer 73230c32551f4370926dd72ed159266e ({username='slave'} at 127.20.120.1:55791)
I20250814 07:58:38.698518 24745 tablet_copy_service.cc:161] P 7b744475f7264e8b82e4fd5b665570dc: Beginning new tablet copy session on tablet a34062205d144210a396ae4b41bd9782 from peer 73230c32551f4370926dd72ed159266e at {username='slave'} at 127.20.120.1:55791: session id = 73230c32551f4370926dd72ed159266e-a34062205d144210a396ae4b41bd9782
I20250814 07:58:38.702447 24745 tablet_copy_source_session.cc:215] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Tablet Copy: opened 0 blocks and 1 log segments
I20250814 07:58:38.704953 24795 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a34062205d144210a396ae4b41bd9782. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:58:38.713982 24795 tablet_copy_client.cc:806] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: tablet copy: Starting download of 0 data blocks...
I20250814 07:58:38.714475 24795 tablet_copy_client.cc:670] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: tablet copy: Starting download of 1 WAL segments...
I20250814 07:58:38.717168 24795 tablet_copy_client.cc:538] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250814 07:58:38.721683 24795 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Bootstrap starting.
I20250814 07:58:38.728662 24786 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:38.729315 24786 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap complete.
I20250814 07:58:38.729790 24786 ts_tablet_manager.cc:1397] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Time spent bootstrapping tablet: real 0.098s user 0.079s sys 0.020s
I20250814 07:58:38.731802 24786 raft_consensus.cc:357] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } }
I20250814 07:58:38.732412 24786 raft_consensus.cc:738] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 38b524e02a3b42ee8cfc21431efded38, State: Initialized, Role: LEARNER
I20250814 07:58:38.732898 24786 consensus_queue.cc:260] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } }
I20250814 07:58:38.735188 24786 ts_tablet_manager.cc:1428] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20250814 07:58:38.736928 24745 tablet_copy_service.cc:342] P 7b744475f7264e8b82e4fd5b665570dc: Request end of tablet copy session 38b524e02a3b42ee8cfc21431efded38-a34062205d144210a396ae4b41bd9782 received from {username='slave'} at 127.20.120.2:35645
I20250814 07:58:38.737485 24745 tablet_copy_service.cc:434] P 7b744475f7264e8b82e4fd5b665570dc: ending tablet copy session 38b524e02a3b42ee8cfc21431efded38-a34062205d144210a396ae4b41bd9782 on tablet a34062205d144210a396ae4b41bd9782 with peer 38b524e02a3b42ee8cfc21431efded38
I20250814 07:58:38.762308 24791 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:38.762858 24791 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap complete.
I20250814 07:58:38.763232 24791 ts_tablet_manager.cc:1397] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Time spent bootstrapping tablet: real 0.111s user 0.097s sys 0.009s
I20250814 07:58:38.764750 24791 raft_consensus.cc:357] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: NON_VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: true } }
I20250814 07:58:38.765257 24791 raft_consensus.cc:738] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 7b744475f7264e8b82e4fd5b665570dc, State: Initialized, Role: LEARNER
I20250814 07:58:38.765621 24791 consensus_queue.cc:260] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: NON_VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: true } }
I20250814 07:58:38.766858 24791 ts_tablet_manager.cc:1428] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20250814 07:58:38.768152 24430 tablet_copy_service.cc:342] P 73230c32551f4370926dd72ed159266e: Request end of tablet copy session 7b744475f7264e8b82e4fd5b665570dc-967c771199494673af94bd0b45841b7d received from {username='slave'} at 127.20.120.3:48835
I20250814 07:58:38.768535 24430 tablet_copy_service.cc:434] P 73230c32551f4370926dd72ed159266e: ending tablet copy session 7b744475f7264e8b82e4fd5b665570dc-967c771199494673af94bd0b45841b7d on tablet 967c771199494673af94bd0b45841b7d with peer 7b744475f7264e8b82e4fd5b665570dc
I20250814 07:58:38.795282 24795 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:38.796162 24795 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Bootstrap complete.
I20250814 07:58:38.796566 24795 ts_tablet_manager.cc:1397] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Time spent bootstrapping tablet: real 0.075s user 0.056s sys 0.016s
I20250814 07:58:38.798139 24795 raft_consensus.cc:357] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } }
I20250814 07:58:38.798560 24795 raft_consensus.cc:738] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 73230c32551f4370926dd72ed159266e, State: Initialized, Role: LEARNER
I20250814 07:58:38.798993 24795 consensus_queue.cc:260] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: NON_VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: true } }
I20250814 07:58:38.800364 24795 ts_tablet_manager.cc:1428] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250814 07:58:38.801574 24745 tablet_copy_service.cc:342] P 7b744475f7264e8b82e4fd5b665570dc: Request end of tablet copy session 73230c32551f4370926dd72ed159266e-a34062205d144210a396ae4b41bd9782 received from {username='slave'} at 127.20.120.1:55791
I20250814 07:58:38.801921 24745 tablet_copy_service.cc:434] P 7b744475f7264e8b82e4fd5b665570dc: ending tablet copy session 73230c32551f4370926dd72ed159266e-a34062205d144210a396ae4b41bd9782 on tablet a34062205d144210a396ae4b41bd9782 with peer 73230c32551f4370926dd72ed159266e
I20250814 07:58:39.032739 24554 raft_consensus.cc:1215] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250814 07:58:39.051735 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 73230c32551f4370926dd72ed159266e to finish bootstrapping
I20250814 07:58:39.063680 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 38b524e02a3b42ee8cfc21431efded38 to finish bootstrapping
I20250814 07:58:39.074697 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 7b744475f7264e8b82e4fd5b665570dc to finish bootstrapping
I20250814 07:58:39.215901 24410 raft_consensus.cc:1215] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
I20250814 07:58:39.231052 24724 raft_consensus.cc:1215] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250814 07:58:39.316735 24534 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 07:58:39.319682 24390 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 07:58:39.323868 24705 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
8475732120104374b7ec894445ee12c6 | 127.20.120.62:40161 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.120.20:41957 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
38b524e02a3b42ee8cfc21431efded38 | 127.20.120.2:46099 | HEALTHY | <none> | 1 | 0
73230c32551f4370926dd72ed159266e | 127.20.120.1:46459 | HEALTHY | <none> | 1 | 0
7b744475f7264e8b82e4fd5b665570dc | 127.20.120.3:39755 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.120.1 | experimental | 127.20.120.1:46459
local_ip_for_outbound_sockets | 127.20.120.2 | experimental | 127.20.120.2:46099
local_ip_for_outbound_sockets | 127.20.120.3 | experimental | 127.20.120.3:39755
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.20.120.1:46459
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.20.120.2:46099
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.20.120.3:39755
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.120.20:41957 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250814 07:58:39.548151 20960 log_verifier.cc:126] Checking tablet 056a8fdaa76046ad8db0b7f717d239f8
I20250814 07:58:39.582789 20960 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 056a8fdaa76046ad8db0b7f717d239f8
I20250814 07:58:39.583133 20960 log_verifier.cc:126] Checking tablet 967c771199494673af94bd0b45841b7d
I20250814 07:58:39.633304 24800 raft_consensus.cc:1062] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: attempting to promote NON_VOTER 38b524e02a3b42ee8cfc21431efded38 to VOTER
I20250814 07:58:39.635540 24800 consensus_queue.cc:237] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:39.641160 24554 raft_consensus.cc:1273] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 LEARNER]: Refusing update from remote peer 7b744475f7264e8b82e4fd5b665570dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250814 07:58:39.642163 24410 raft_consensus.cc:1273] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 LEARNER]: Refusing update from remote peer 7b744475f7264e8b82e4fd5b665570dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250814 07:58:39.642650 24799 consensus_queue.cc:1035] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250814 07:58:39.643571 24800 consensus_queue.cc:1035] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20250814 07:58:39.649583 24799 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:39.651666 24410 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 LEARNER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:39.663048 24799 raft_consensus.cc:1062] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: attempting to promote NON_VOTER 73230c32551f4370926dd72ed159266e to VOTER
I20250814 07:58:39.664445 24799 consensus_queue.cc:237] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:39.664619 24266 catalog_manager.cc:5582] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e reported cstate change: config changed from index 9 to 10, 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7b744475f7264e8b82e4fd5b665570dc" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:39.666574 24554 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: NON_VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: true } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:39.677268 24410 raft_consensus.cc:1273] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 LEARNER]: Refusing update from remote peer 7b744475f7264e8b82e4fd5b665570dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250814 07:58:39.678565 24799 consensus_queue.cc:1035] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250814 07:58:39.684926 24800 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 73230c32551f4370926dd72ed159266e (127.20.120.1) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:39.687383 20960 log_verifier.cc:177] Verified matching terms for 11 ops in tablet 967c771199494673af94bd0b45841b7d
I20250814 07:58:39.686524 24410 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 73230c32551f4370926dd72ed159266e (127.20.120.1) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:39.687752 20960 log_verifier.cc:126] Checking tablet a34062205d144210a396ae4b41bd9782
I20250814 07:58:39.695284 24553 raft_consensus.cc:1273] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Refusing update from remote peer 7b744475f7264e8b82e4fd5b665570dc: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250814 07:58:39.702785 24778 consensus_queue.cc:1035] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250814 07:58:39.712625 24812 raft_consensus.cc:1062] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: attempting to promote NON_VOTER 7b744475f7264e8b82e4fd5b665570dc to VOTER
I20250814 07:58:39.711907 24266 catalog_manager.cc:5582] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e reported cstate change: config changed from index 10 to 11, 73230c32551f4370926dd72ed159266e (127.20.120.1) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "7b744475f7264e8b82e4fd5b665570dc" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:39.712903 24554 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, 73230c32551f4370926dd72ed159266e (127.20.120.1) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:39.716501 24812 consensus_queue.cc:237] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
I20250814 07:58:39.726912 24724 raft_consensus.cc:1273] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc [term 2 LEARNER]: Refusing update from remote peer 73230c32551f4370926dd72ed159266e: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250814 07:58:39.728906 24554 raft_consensus.cc:1273] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Refusing update from remote peer 73230c32551f4370926dd72ed159266e: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250814 07:58:39.728824 24790 consensus_queue.cc:1035] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250814 07:58:39.730327 24790 consensus_queue.cc:1035] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250814 07:58:39.737241 24812 raft_consensus.cc:2953] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } } }
I20250814 07:58:39.738937 24554 raft_consensus.cc:2953] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } } }
I20250814 07:58:39.739133 24724 raft_consensus.cc:2953] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } } }
I20250814 07:58:39.751022 24267 catalog_manager.cc:5582] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e reported cstate change: config changed from index 11 to 12, 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "73230c32551f4370926dd72ed159266e" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250814 07:58:39.785856 20960 log_verifier.cc:177] Verified matching terms for 11 ops in tablet a34062205d144210a396ae4b41bd9782
I20250814 07:58:39.786187 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 24234
I20250814 07:58:39.814169 20960 minidump.cc:252] Setting minidump size limit to 20M
I20250814 07:58:39.815487 20960 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:39.816421 20960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:39.825887 24843 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:39.827255 24846 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:39.826279 24844 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:39.925566 20960 server_base.cc:1047] running on GCE node
I20250814 07:58:39.926517 20960 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250814 07:58:39.926717 20960 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250814 07:58:39.926844 20960 hybrid_clock.cc:648] HybridClock initialized: now 1755158319926830 us; error 0 us; skew 500 ppm
I20250814 07:58:39.927407 20960 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:39.930076 20960 webserver.cc:480] Webserver started at http://0.0.0.0:46279/ using document root <none> and password file <none>
I20250814 07:58:39.930783 20960 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:39.930954 20960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:39.935652 20960 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20250814 07:58:39.938912 24851 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:39.939673 20960 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20250814 07:58:39.939972 20960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8475732120104374b7ec894445ee12c6"
format_stamp: "Formatted at 2025-08-14 07:58:17 on dist-test-slave-87c1"
I20250814 07:58:39.941663 20960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:39.958050 20960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:39.959590 20960 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:39.960050 20960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:39.971660 20960 sys_catalog.cc:263] Verifying existing consensus state
W20250814 07:58:39.977185 20960 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.20.120.62:40161 exists but no master address supplied!
I20250814 07:58:39.979158 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Bootstrap starting.
I20250814 07:58:40.030522 20960 log.cc:826] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:40.094337 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:40.095283 20960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Bootstrap complete.
I20250814 07:58:40.113265 20960 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:40.113792 20960 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8475732120104374b7ec894445ee12c6, State: Initialized, Role: FOLLOWER
I20250814 07:58:40.114516 20960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:40.114977 20960 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:40.115212 20960 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:40.115516 20960 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 3 FOLLOWER]: Advancing to term 4
I20250814 07:58:40.123123 20960 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:40.123878 20960 leader_election.cc:304] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8475732120104374b7ec894445ee12c6; no voters:
I20250814 07:58:40.124899 20960 leader_election.cc:290] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [CANDIDATE]: Term 4 election: Requested vote from peers
I20250814 07:58:40.125198 24858 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 4 FOLLOWER]: Leader election won for term 4
I20250814 07:58:40.126482 24858 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 4 LEADER]: Becoming Leader. State: Replica: 8475732120104374b7ec894445ee12c6, State: Running, Role: LEADER
I20250814 07:58:40.127233 24858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:40.135222 24860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8475732120104374b7ec894445ee12c6. Latest consensus state: current_term: 4 leader_uuid: "8475732120104374b7ec894445ee12c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } } }
I20250814 07:58:40.135681 24860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: This master's current role is: LEADER
I20250814 07:58:40.135543 24859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "8475732120104374b7ec894445ee12c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } } }
I20250814 07:58:40.136144 24859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: This master's current role is: LEADER
I20250814 07:58:40.160589 20960 tablet_replica.cc:331] stopping tablet replica
I20250814 07:58:40.161106 20960 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 4 LEADER]: Raft consensus shutting down.
I20250814 07:58:40.161473 20960 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 4 FOLLOWER]: Raft consensus is shut down!
I20250814 07:58:40.163762 20960 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250814 07:58:40.164222 20960 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250814 07:58:40.194003 20960 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
W20250814 07:58:40.339184 24770 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:40161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:40161: connect: Connection refused (error 111)
W20250814 07:58:40.775147 24455 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:40161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:40161: connect: Connection refused (error 111)
W20250814 07:58:40.787015 24604 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:40161 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:40161: connect: Connection refused (error 111)
I20250814 07:58:45.003366 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 24305
I20250814 07:58:45.028847 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 24459
I20250814 07:58:45.055109 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 24608
I20250814 07:58:45.082669 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:40161
--webserver_interface=127.20.120.62
--webserver_port=45103
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:40161 with env {}
W20250814 07:58:45.366735 24937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:45.367280 24937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:45.367728 24937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:45.397163 24937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:58:45.397457 24937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:45.397701 24937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:58:45.397941 24937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:58:45.431012 24937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:40161
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:40161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=45103
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:45.432220 24937 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:45.433749 24937 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:45.443249 24943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:46.847678 24942 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 24937
W20250814 07:58:47.093556 24942 kernel_stack_watchdog.cc:198] Thread 24937 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 07:58:45.445037 24944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:47.094380 24937 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.650s user 0.609s sys 1.037s
W20250814 07:58:47.094846 24937 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.651s user 0.609s sys 1.037s
W20250814 07:58:47.096386 24946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:47.098011 24945 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1650 milliseconds
I20250814 07:58:47.098063 24937 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:47.099264 24937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:47.101729 24937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:47.103072 24937 hybrid_clock.cc:648] HybridClock initialized: now 1755158327103040 us; error 32 us; skew 500 ppm
I20250814 07:58:47.103863 24937 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:47.109969 24937 webserver.cc:480] Webserver started at http://127.20.120.62:45103/ using document root <none> and password file <none>
I20250814 07:58:47.110870 24937 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:47.111084 24937 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:47.118520 24937 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250814 07:58:47.123178 24953 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:47.124099 24937 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250814 07:58:47.124406 24937 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "8475732120104374b7ec894445ee12c6"
format_stamp: "Formatted at 2025-08-14 07:58:17 on dist-test-slave-87c1"
I20250814 07:58:47.126307 24937 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:47.188723 24937 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:47.190021 24937 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:47.190402 24937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:47.255241 24937 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:40161
I20250814 07:58:47.255306 25004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:40161 every 8 connection(s)
I20250814 07:58:47.257956 24937 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 07:58:47.266721 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 24937
I20250814 07:58:47.267911 25005 sys_catalog.cc:263] Verifying existing consensus state
I20250814 07:58:47.268687 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:46459
--local_ip_for_outbound_sockets=127.20.120.1
--tserver_master_addrs=127.20.120.62:40161
--webserver_port=46029
--webserver_interface=127.20.120.1
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:58:47.275148 25005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Bootstrap starting.
I20250814 07:58:47.285820 25005 log.cc:826] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:47.356072 25005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:47.356792 25005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Bootstrap complete.
I20250814 07:58:47.373723 25005 raft_consensus.cc:357] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:47.375519 25005 raft_consensus.cc:738] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8475732120104374b7ec894445ee12c6, State: Initialized, Role: FOLLOWER
I20250814 07:58:47.376161 25005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:47.376582 25005 raft_consensus.cc:397] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:47.376824 25005 raft_consensus.cc:491] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:47.377146 25005 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 5 FOLLOWER]: Advancing to term 6
I20250814 07:58:47.381651 25005 raft_consensus.cc:513] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:47.382201 25005 leader_election.cc:304] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8475732120104374b7ec894445ee12c6; no voters:
I20250814 07:58:47.384353 25005 leader_election.cc:290] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [CANDIDATE]: Term 6 election: Requested vote from peers
I20250814 07:58:47.384729 25009 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 6 FOLLOWER]: Leader election won for term 6
I20250814 07:58:47.387787 25009 raft_consensus.cc:695] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [term 6 LEADER]: Becoming Leader. State: Replica: 8475732120104374b7ec894445ee12c6, State: Running, Role: LEADER
I20250814 07:58:47.388610 25009 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } }
I20250814 07:58:47.389065 25005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:58:47.398305 25010 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "8475732120104374b7ec894445ee12c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } } }
I20250814 07:58:47.398316 25011 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8475732120104374b7ec894445ee12c6. Latest consensus state: current_term: 6 leader_uuid: "8475732120104374b7ec894445ee12c6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8475732120104374b7ec894445ee12c6" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 40161 } } }
I20250814 07:58:47.399070 25010 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: This master's current role is: LEADER
I20250814 07:58:47.399160 25011 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6 [sys.catalog]: This master's current role is: LEADER
I20250814 07:58:47.410470 25016 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:58:47.424803 25016 catalog_manager.cc:671] Loaded metadata for table TestTable [id=7daa466fbb1649fc932a146e8692db4d]
I20250814 07:58:47.427183 25016 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=b565d9c2cccc451f9a8999179837e866]
I20250814 07:58:47.428742 25016 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=c510acfde0144ac08f7d263d93711de5]
I20250814 07:58:47.436762 25016 tablet_loader.cc:96] loaded metadata for tablet 056a8fdaa76046ad8db0b7f717d239f8 (table TestTable2 [id=c510acfde0144ac08f7d263d93711de5])
I20250814 07:58:47.438236 25016 tablet_loader.cc:96] loaded metadata for tablet 967c771199494673af94bd0b45841b7d (table TestTable1 [id=b565d9c2cccc451f9a8999179837e866])
I20250814 07:58:47.439384 25016 tablet_loader.cc:96] loaded metadata for tablet a34062205d144210a396ae4b41bd9782 (table TestTable [id=7daa466fbb1649fc932a146e8692db4d])
I20250814 07:58:47.440667 25016 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:58:47.447072 25016 catalog_manager.cc:1261] Loaded cluster ID: 4165327691984017b938ffa38db7cd24
I20250814 07:58:47.447340 25016 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:58:47.455572 25016 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:58:47.461728 25016 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 8475732120104374b7ec894445ee12c6: Loaded TSK: 0
I20250814 07:58:47.463366 25016 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250814 07:58:47.586256 25007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:47.586709 25007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:47.587184 25007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:47.616545 25007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:47.617298 25007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 07:58:47.651216 25007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:46459
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=46029
--tserver_master_addrs=127.20.120.62:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:47.652412 25007 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:47.653934 25007 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:47.665447 25033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:47.666697 25034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:49.057195 25035 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1390 milliseconds
W20250814 07:58:49.057941 25036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:49.059010 25007 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.392s user 0.393s sys 0.955s
W20250814 07:58:49.059376 25007 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.393s user 0.393s sys 0.955s
I20250814 07:58:49.059667 25007 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:49.061272 25007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:49.065569 25007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:49.067086 25007 hybrid_clock.cc:648] HybridClock initialized: now 1755158329067004 us; error 80 us; skew 500 ppm
I20250814 07:58:49.068265 25007 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:49.077800 25007 webserver.cc:480] Webserver started at http://127.20.120.1:46029/ using document root <none> and password file <none>
I20250814 07:58:49.079200 25007 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:49.079505 25007 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:49.090973 25007 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.008s sys 0.000s
I20250814 07:58:49.097466 25043 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:49.098737 25007 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250814 07:58:49.099172 25007 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "73230c32551f4370926dd72ed159266e"
format_stamp: "Formatted at 2025-08-14 07:58:19 on dist-test-slave-87c1"
I20250814 07:58:49.102074 25007 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:49.187758 25007 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:49.189733 25007 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:49.190368 25007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:49.193866 25007 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:49.200533 25050 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250814 07:58:49.211163 25007 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250814 07:58:49.211355 25007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.002s
I20250814 07:58:49.211580 25007 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250814 07:58:49.216557 25050 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Bootstrap starting.
I20250814 07:58:49.221264 25007 ts_tablet_manager.cc:610] Registered 2 tablets
I20250814 07:58:49.221522 25007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.010s sys 0.000s
I20250814 07:58:49.291344 25050 log.cc:826] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:49.421531 25050 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:49.422574 25050 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Bootstrap complete.
I20250814 07:58:49.424365 25050 ts_tablet_manager.cc:1397] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Time spent bootstrapping tablet: real 0.208s user 0.162s sys 0.041s
I20250814 07:58:49.440660 25007 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:46459
I20250814 07:58:49.440923 25157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:46459 every 8 connection(s)
I20250814 07:58:49.443181 25007 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 07:58:49.446619 25050 raft_consensus.cc:357] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:49.449812 25050 raft_consensus.cc:738] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73230c32551f4370926dd72ed159266e, State: Initialized, Role: FOLLOWER
I20250814 07:58:49.451184 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 25007
I20250814 07:58:49.451675 25050 consensus_queue.cc:260] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:49.455579 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.2:46099
--local_ip_for_outbound_sockets=127.20.120.2
--tserver_master_addrs=127.20.120.62:40161
--webserver_port=46253
--webserver_interface=127.20.120.2
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:58:49.456219 25050 ts_tablet_manager.cc:1428] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e: Time spent starting tablet: real 0.032s user 0.025s sys 0.003s
I20250814 07:58:49.456962 25050 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Bootstrap starting.
I20250814 07:58:49.500020 25158 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40161
I20250814 07:58:49.500483 25158 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:49.501741 25158 heartbeater.cc:507] Master 127.20.120.62:40161 requested a full tablet report, sending...
I20250814 07:58:49.506705 24970 ts_manager.cc:194] Registered new tserver with Master: 73230c32551f4370926dd72ed159266e (127.20.120.1:46459)
I20250814 07:58:49.511183 24970 catalog_manager.cc:5582] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e reported cstate change: config changed from index -1 to 11, term changed from 0 to 2, VOTER 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) added, VOTER 73230c32551f4370926dd72ed159266e (127.20.120.1) added, VOTER 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) added. New cstate: current_term: 2 committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:49.563063 24970 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:55871
I20250814 07:58:49.567440 25158 heartbeater.cc:499] Master 127.20.120.62:40161 was elected leader, sending a full tablet report...
I20250814 07:58:49.603752 25050 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:49.604430 25050 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Bootstrap complete.
I20250814 07:58:49.605620 25050 ts_tablet_manager.cc:1397] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Time spent bootstrapping tablet: real 0.149s user 0.116s sys 0.019s
I20250814 07:58:49.607086 25050 raft_consensus.cc:357] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
I20250814 07:58:49.607517 25050 raft_consensus.cc:738] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73230c32551f4370926dd72ed159266e, State: Initialized, Role: FOLLOWER
I20250814 07:58:49.608014 25050 consensus_queue.cc:260] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
I20250814 07:58:49.609601 25050 ts_tablet_manager.cc:1428] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250814 07:58:49.815338 25163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:49.815826 25163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:49.816311 25163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:49.845590 25163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:49.846342 25163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.2
I20250814 07:58:49.877310 25163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.2:46099
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.120.2
--webserver_port=46253
--tserver_master_addrs=127.20.120.62:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.2
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:49.878499 25163 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:49.879945 25163 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:49.891428 25174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:50.839208 25181 raft_consensus.cc:491] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:58:50.840415 25181 raft_consensus.cc:513] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
W20250814 07:58:50.851716 25046 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.3:39755: connect: Connection refused (error 111)
I20250814 07:58:50.854025 25181 leader_election.cc:290] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755), 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099)
W20250814 07:58:50.861429 25046 leader_election.cc:336] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755): Network error: Client connection negotiation failed: client connection to 127.20.120.3:39755: connect: Connection refused (error 111)
W20250814 07:58:50.862712 25046 leader_election.cc:336] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099): Network error: Client connection negotiation failed: client connection to 127.20.120.2:46099: connect: Connection refused (error 111)
I20250814 07:58:50.863377 25046 leader_election.cc:304] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73230c32551f4370926dd72ed159266e; no voters: 38b524e02a3b42ee8cfc21431efded38, 7b744475f7264e8b82e4fd5b665570dc
I20250814 07:58:50.864529 25181 raft_consensus.cc:2747] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250814 07:58:50.866242 25181 raft_consensus.cc:491] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:58:50.866891 25181 raft_consensus.cc:513] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
I20250814 07:58:50.869469 25181 leader_election.cc:290] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099), 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755)
W20250814 07:58:50.885283 25046 leader_election.cc:336] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099): Network error: Client connection negotiation failed: client connection to 127.20.120.2:46099: connect: Connection refused (error 111)
W20250814 07:58:50.886134 25046 leader_election.cc:336] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755): Network error: Client connection negotiation failed: client connection to 127.20.120.3:39755: connect: Connection refused (error 111)
I20250814 07:58:50.886701 25046 leader_election.cc:304] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 73230c32551f4370926dd72ed159266e; no voters: 38b524e02a3b42ee8cfc21431efded38, 7b744475f7264e8b82e4fd5b665570dc
I20250814 07:58:50.889107 25181 raft_consensus.cc:2747] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250814 07:58:49.892884 25175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:51.222543 25177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:51.225575 25176 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1331 milliseconds
W20250814 07:58:51.226742 25163 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.334s user 0.445s sys 0.817s
W20250814 07:58:51.226995 25163 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.334s user 0.445s sys 0.817s
I20250814 07:58:51.227185 25163 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:51.228212 25163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:51.230401 25163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:51.231737 25163 hybrid_clock.cc:648] HybridClock initialized: now 1755158331231699 us; error 38 us; skew 500 ppm
I20250814 07:58:51.232445 25163 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:51.239405 25163 webserver.cc:480] Webserver started at http://127.20.120.2:46253/ using document root <none> and password file <none>
I20250814 07:58:51.240301 25163 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:51.240494 25163 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:51.247969 25163 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.005s
I20250814 07:58:51.253203 25189 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:51.254408 25163 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250814 07:58:51.254753 25163 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "38b524e02a3b42ee8cfc21431efded38"
format_stamp: "Formatted at 2025-08-14 07:58:21 on dist-test-slave-87c1"
I20250814 07:58:51.257246 25163 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:51.351888 25163 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:51.353363 25163 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:51.353783 25163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:51.356144 25163 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:51.361687 25196 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250814 07:58:51.376452 25163 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250814 07:58:51.376694 25163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.000s sys 0.002s
I20250814 07:58:51.376938 25163 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250814 07:58:51.381899 25196 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap starting.
I20250814 07:58:51.387419 25163 ts_tablet_manager.cc:610] Registered 3 tablets
I20250814 07:58:51.387589 25163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.011s user 0.009s sys 0.000s
I20250814 07:58:51.430104 25196 log.cc:826] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:51.514372 25196 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:51.515106 25196 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap complete.
I20250814 07:58:51.516510 25196 ts_tablet_manager.cc:1397] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Time spent bootstrapping tablet: real 0.135s user 0.101s sys 0.030s
I20250814 07:58:51.530966 25196 raft_consensus.cc:357] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:51.532992 25196 raft_consensus.cc:738] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b524e02a3b42ee8cfc21431efded38, State: Initialized, Role: FOLLOWER
I20250814 07:58:51.533864 25196 consensus_queue.cc:260] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:51.537518 25196 ts_tablet_manager.cc:1428] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Time spent starting tablet: real 0.021s user 0.021s sys 0.000s
I20250814 07:58:51.538121 25196 tablet_bootstrap.cc:492] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap starting.
I20250814 07:58:51.567472 25163 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.2:46099
I20250814 07:58:51.567565 25304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.2:46099 every 8 connection(s)
I20250814 07:58:51.569787 25163 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 07:58:51.579890 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 25163
I20250814 07:58:51.581542 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.3:39755
--local_ip_for_outbound_sockets=127.20.120.3
--tserver_master_addrs=127.20.120.62:40161
--webserver_port=35023
--webserver_interface=127.20.120.3
--builtin_ntp_servers=127.20.120.20:41957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250814 07:58:51.588865 25305 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40161
I20250814 07:58:51.589349 25305 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:51.590420 25305 heartbeater.cc:507] Master 127.20.120.62:40161 requested a full tablet report, sending...
I20250814 07:58:51.594699 24969 ts_manager.cc:194] Registered new tserver with Master: 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099)
I20250814 07:58:51.599870 24969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.2:45511
I20250814 07:58:51.629971 25196 tablet_bootstrap.cc:492] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:51.630590 25196 tablet_bootstrap.cc:492] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Bootstrap complete.
I20250814 07:58:51.631742 25196 ts_tablet_manager.cc:1397] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Time spent bootstrapping tablet: real 0.094s user 0.059s sys 0.031s
I20250814 07:58:51.633307 25196 raft_consensus.cc:357] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:51.633663 25196 raft_consensus.cc:738] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b524e02a3b42ee8cfc21431efded38, State: Initialized, Role: FOLLOWER
I20250814 07:58:51.634132 25196 consensus_queue.cc:260] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:51.634490 25196 raft_consensus.cc:397] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:58:51.634725 25196 raft_consensus.cc:491] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:58:51.634989 25196 raft_consensus.cc:3058] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Advancing to term 3
I20250814 07:58:51.639899 25196 raft_consensus.cc:513] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:51.640487 25196 leader_election.cc:304] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38b524e02a3b42ee8cfc21431efded38; no voters:
I20250814 07:58:51.640950 25196 leader_election.cc:290] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [CANDIDATE]: Term 3 election: Requested vote from peers
I20250814 07:58:51.641254 25286 raft_consensus.cc:2802] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 3 FOLLOWER]: Leader election won for term 3
I20250814 07:58:51.643910 25305 heartbeater.cc:499] Master 127.20.120.62:40161 was elected leader, sending a full tablet report...
I20250814 07:58:51.644443 25196 ts_tablet_manager.cc:1428] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20250814 07:58:51.645241 25196 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Bootstrap starting.
I20250814 07:58:51.645475 25286 raft_consensus.cc:695] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [term 3 LEADER]: Becoming Leader. State: Replica: 38b524e02a3b42ee8cfc21431efded38, State: Running, Role: LEADER
I20250814 07:58:51.646279 25286 consensus_queue.cc:237] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } }
I20250814 07:58:51.658159 24969 catalog_manager.cc:5582] T 056a8fdaa76046ad8db0b7f717d239f8 P 38b524e02a3b42ee8cfc21431efded38 reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "38b524e02a3b42ee8cfc21431efded38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } health_report { overall_health: HEALTHY } } }
I20250814 07:58:51.746479 25196 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:51.747120 25196 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Bootstrap complete.
I20250814 07:58:51.748215 25196 ts_tablet_manager.cc:1397] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Time spent bootstrapping tablet: real 0.103s user 0.092s sys 0.007s
I20250814 07:58:51.749951 25196 raft_consensus.cc:357] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
I20250814 07:58:51.750368 25196 raft_consensus.cc:738] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38b524e02a3b42ee8cfc21431efded38, State: Initialized, Role: FOLLOWER
I20250814 07:58:51.750893 25196 consensus_queue.cc:260] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
I20250814 07:58:51.752424 25196 ts_tablet_manager.cc:1428] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250814 07:58:51.895833 25310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:58:51.896286 25310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:58:51.896768 25310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:58:51.926194 25310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:58:51.926923 25310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.3
I20250814 07:58:51.957742 25310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.3:39755
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.120.3
--webserver_port=35023
--tserver_master_addrs=127.20.120.62:40161
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.3
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:58:51.959012 25310 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:58:51.960490 25310 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:58:51.971096 25325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:58:52.547820 25331 raft_consensus.cc:491] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:58:52.548805 25331 raft_consensus.cc:513] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:52.557934 25331 leader_election.cc:290] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755), 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099)
W20250814 07:58:52.587345 25046 leader_election.cc:336] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755): Network error: Client connection negotiation failed: client connection to 127.20.120.3:39755: connect: Connection refused (error 111)
I20250814 07:58:52.605543 25259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34062205d144210a396ae4b41bd9782" candidate_uuid: "73230c32551f4370926dd72ed159266e" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "38b524e02a3b42ee8cfc21431efded38" is_pre_election: true
I20250814 07:58:52.606554 25259 raft_consensus.cc:2466] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73230c32551f4370926dd72ed159266e in term 2.
I20250814 07:58:52.608721 25046 leader_election.cc:304] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38b524e02a3b42ee8cfc21431efded38, 73230c32551f4370926dd72ed159266e; no voters: 7b744475f7264e8b82e4fd5b665570dc
I20250814 07:58:52.609995 25331 raft_consensus.cc:2802] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250814 07:58:52.610399 25331 raft_consensus.cc:491] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 07:58:52.610786 25331 raft_consensus.cc:3058] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Advancing to term 3
I20250814 07:58:52.623709 25331 raft_consensus.cc:513] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:52.636246 25259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a34062205d144210a396ae4b41bd9782" candidate_uuid: "73230c32551f4370926dd72ed159266e" candidate_term: 3 candidate_status { last_received { term: 2 index: 11 } } ignore_live_leader: false dest_uuid: "38b524e02a3b42ee8cfc21431efded38"
I20250814 07:58:52.636933 25259 raft_consensus.cc:3058] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Advancing to term 3
W20250814 07:58:52.638937 25046 leader_election.cc:336] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755): Network error: Client connection negotiation failed: client connection to 127.20.120.3:39755: connect: Connection refused (error 111)
I20250814 07:58:52.645923 25259 raft_consensus.cc:2466] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73230c32551f4370926dd72ed159266e in term 3.
I20250814 07:58:52.647234 25046 leader_election.cc:304] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38b524e02a3b42ee8cfc21431efded38, 73230c32551f4370926dd72ed159266e; no voters: 7b744475f7264e8b82e4fd5b665570dc
I20250814 07:58:52.661834 25331 leader_election.cc:290] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 election: Requested vote from peers 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755), 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099)
I20250814 07:58:52.662401 25336 raft_consensus.cc:2802] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 3 FOLLOWER]: Leader election won for term 3
I20250814 07:58:52.663652 25336 raft_consensus.cc:695] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 3 LEADER]: Becoming Leader. State: Replica: 73230c32551f4370926dd72ed159266e, State: Running, Role: LEADER
I20250814 07:58:52.666653 25336 consensus_queue.cc:237] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:52.689955 24969 catalog_manager.cc:5582] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e reported cstate change: term changed from 2 to 3, leader changed from <none> to 73230c32551f4370926dd72ed159266e (127.20.120.1). New cstate: current_term: 3 leader_uuid: "73230c32551f4370926dd72ed159266e" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20250814 07:58:52.745631 25336 raft_consensus.cc:491] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:58:52.746196 25336 raft_consensus.cc:513] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
I20250814 07:58:52.748492 25336 leader_election.cc:290] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099), 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755)
I20250814 07:58:52.753275 25259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "967c771199494673af94bd0b45841b7d" candidate_uuid: "73230c32551f4370926dd72ed159266e" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "38b524e02a3b42ee8cfc21431efded38" is_pre_election: true
I20250814 07:58:52.753999 25259 raft_consensus.cc:2466] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 73230c32551f4370926dd72ed159266e in term 2.
I20250814 07:58:52.757624 25046 leader_election.cc:304] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38b524e02a3b42ee8cfc21431efded38, 73230c32551f4370926dd72ed159266e; no voters:
I20250814 07:58:52.759720 25336 raft_consensus.cc:2802] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250814 07:58:52.768301 25336 raft_consensus.cc:491] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 07:58:52.769173 25336 raft_consensus.cc:3058] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 2 FOLLOWER]: Advancing to term 3
I20250814 07:58:52.778118 25336 raft_consensus.cc:513] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
W20250814 07:58:52.768949 25046 leader_election.cc:336] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755): Network error: Client connection negotiation failed: client connection to 127.20.120.3:39755: connect: Connection refused (error 111)
I20250814 07:58:52.782346 25259 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "967c771199494673af94bd0b45841b7d" candidate_uuid: "73230c32551f4370926dd72ed159266e" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "38b524e02a3b42ee8cfc21431efded38"
I20250814 07:58:52.783025 25259 raft_consensus.cc:3058] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 2 FOLLOWER]: Advancing to term 3
I20250814 07:58:52.791467 25259 raft_consensus.cc:2466] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 73230c32551f4370926dd72ed159266e in term 3.
W20250814 07:58:52.799247 25046 leader_election.cc:336] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755): Network error: Client connection negotiation failed: client connection to 127.20.120.3:39755: connect: Connection refused (error 111)
I20250814 07:58:52.801136 25046 leader_election.cc:304] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38b524e02a3b42ee8cfc21431efded38, 73230c32551f4370926dd72ed159266e; no voters: 7b744475f7264e8b82e4fd5b665570dc
I20250814 07:58:52.802120 25331 raft_consensus.cc:2802] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 3 FOLLOWER]: Leader election won for term 3
I20250814 07:58:52.800004 25336 leader_election.cc:290] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [CANDIDATE]: Term 3 election: Requested vote from peers 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099), 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755)
I20250814 07:58:52.813653 25331 raft_consensus.cc:695] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [term 3 LEADER]: Becoming Leader. State: Replica: 73230c32551f4370926dd72ed159266e, State: Running, Role: LEADER
I20250814 07:58:52.814703 25331 consensus_queue.cc:237] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
I20250814 07:58:52.831517 24969 catalog_manager.cc:5582] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "73230c32551f4370926dd72ed159266e" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250814 07:58:53.089427 25046 consensus_peers.cc:489] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e -> Peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755): Couldn't send request to peer 7b744475f7264e8b82e4fd5b665570dc. Status: Network error: Client connection negotiation failed: client connection to 127.20.120.3:39755: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20250814 07:58:53.217473 25046 consensus_peers.cc:489] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e -> Peer 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755): Couldn't send request to peer 7b744475f7264e8b82e4fd5b665570dc. Status: Network error: Client connection negotiation failed: client connection to 127.20.120.3:39755: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250814 07:58:53.278291 25259 raft_consensus.cc:1273] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 3 FOLLOWER]: Refusing update from remote peer 73230c32551f4370926dd72ed159266e: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 3 index: 12. (index mismatch)
I20250814 07:58:53.280481 25331 consensus_queue.cc:1035] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250814 07:58:53.283914 25259 raft_consensus.cc:1273] T 967c771199494673af94bd0b45841b7d P 38b524e02a3b42ee8cfc21431efded38 [term 3 FOLLOWER]: Refusing update from remote peer 73230c32551f4370926dd72ed159266e: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250814 07:58:53.285467 25331 consensus_queue.cc:1035] T 967c771199494673af94bd0b45841b7d P 73230c32551f4370926dd72ed159266e [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20250814 07:58:51.971341 25326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:53.309763 25327 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1335 milliseconds
W20250814 07:58:53.310132 25328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:58:53.315469 25310 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.344s user 0.456s sys 0.873s
W20250814 07:58:53.315845 25310 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.345s user 0.457s sys 0.874s
I20250814 07:58:53.316139 25310 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:58:53.317639 25310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:58:53.326246 25310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:58:53.334923 25310 hybrid_clock.cc:648] HybridClock initialized: now 1755158333334858 us; error 51 us; skew 500 ppm
I20250814 07:58:53.336402 25310 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:58:53.373653 25310 webserver.cc:480] Webserver started at http://127.20.120.3:35023/ using document root <none> and password file <none>
I20250814 07:58:53.374702 25310 fs_manager.cc:362] Metadata directory not provided
I20250814 07:58:53.374931 25310 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:58:53.375982 25113 consensus_queue.cc:237] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 3.12, Last appended by leader: 11, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:53.380422 25258 raft_consensus.cc:1273] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 3 FOLLOWER]: Refusing update from remote peer 73230c32551f4370926dd72ed159266e: Log matching property violated. Preceding OpId in replica: term: 3 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250814 07:58:53.381742 25349 consensus_queue.cc:1035] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [LEADER]: Connected to new peer: Peer: permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250814 07:58:53.384266 25310 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20250814 07:58:53.387743 25331 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 3 LEADER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:53.389374 25258 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 3 FOLLOWER]: Committing config change with OpId 3.13: config changed from index 11 to 13, VOTER 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) evicted. New config: { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
I20250814 07:58:53.396041 25360 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:58:53.397253 25310 fs_manager.cc:730] Time spent opening block manager: real 0.010s user 0.008s sys 0.002s
I20250814 07:58:53.398995 24956 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a34062205d144210a396ae4b41bd9782 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250814 07:58:53.397614 25310 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "7b744475f7264e8b82e4fd5b665570dc"
format_stamp: "Formatted at 2025-08-14 07:58:23 on dist-test-slave-87c1"
I20250814 07:58:53.402899 25310 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:58:53.402889 24970 catalog_manager.cc:5582] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 reported cstate change: config changed from index 11 to 13, VOTER 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3) evicted. New cstate: current_term: 3 leader_uuid: "73230c32551f4370926dd72ed159266e" committed_config { opid_index: 13 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } }
W20250814 07:58:53.418467 24970 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet a34062205d144210a396ae4b41bd9782 on TS 7b744475f7264e8b82e4fd5b665570dc: Not found: failed to reset TS proxy: Could not find TS for UUID 7b744475f7264e8b82e4fd5b665570dc
I20250814 07:58:53.422564 25113 consensus_queue.cc:237] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 11, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } }
I20250814 07:58:53.424868 25331 raft_consensus.cc:2953] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 13 to 14, VOTER 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } }
I20250814 07:58:53.433336 24956 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet a34062205d144210a396ae4b41bd9782 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250814 07:58:53.437386 24969 catalog_manager.cc:5582] T a34062205d144210a396ae4b41bd9782 P 73230c32551f4370926dd72ed159266e reported cstate change: config changed from index 13 to 14, VOTER 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2) evicted. New cstate: current_term: 3 leader_uuid: "73230c32551f4370926dd72ed159266e" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250814 07:58:53.466274 25239 tablet_service.cc:1515] Processing DeleteTablet for tablet a34062205d144210a396ae4b41bd9782 with delete_type TABLET_DATA_TOMBSTONED (TS 38b524e02a3b42ee8cfc21431efded38 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:33170
W20250814 07:58:53.471088 24955 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet a34062205d144210a396ae4b41bd9782 on TS 7b744475f7264e8b82e4fd5b665570dc failed: Not found: failed to reset TS proxy: Could not find TS for UUID 7b744475f7264e8b82e4fd5b665570dc
I20250814 07:58:53.477808 25366 tablet_replica.cc:331] stopping tablet replica
I20250814 07:58:53.478962 25366 raft_consensus.cc:2241] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 3 FOLLOWER]: Raft consensus shutting down.
I20250814 07:58:53.479598 25366 raft_consensus.cc:2270] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38 [term 3 FOLLOWER]: Raft consensus is shut down!
I20250814 07:58:53.482867 25366 ts_tablet_manager.cc:1905] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250814 07:58:53.497530 25366 ts_tablet_manager.cc:1918] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.13
I20250814 07:58:53.497895 25366 log.cc:1199] T a34062205d144210a396ae4b41bd9782 P 38b524e02a3b42ee8cfc21431efded38: Deleting WAL directory at /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/wals/a34062205d144210a396ae4b41bd9782
I20250814 07:58:53.499506 24956 catalog_manager.cc:4928] TS 38b524e02a3b42ee8cfc21431efded38 (127.20.120.2:46099): tablet a34062205d144210a396ae4b41bd9782 (table TestTable [id=7daa466fbb1649fc932a146e8692db4d]) successfully deleted
I20250814 07:58:53.502403 25310 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:58:53.503752 25310 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:58:53.504123 25310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:58:53.506456 25310 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:58:53.511665 25372 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250814 07:58:53.522768 25310 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250814 07:58:53.522984 25310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.001s sys 0.001s
I20250814 07:58:53.523213 25310 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250814 07:58:53.527987 25372 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap starting.
I20250814 07:58:53.530503 25310 ts_tablet_manager.cc:610] Registered 2 tablets
I20250814 07:58:53.530704 25310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.006s sys 0.000s
I20250814 07:58:53.585834 25372 log.cc:826] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Log is configured to *not* fsync() on all Append() calls
I20250814 07:58:53.708384 25310 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.3:39755
I20250814 07:58:53.708453 25479 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.3:39755 every 8 connection(s)
I20250814 07:58:53.710094 25372 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:53.711004 25372 tablet_bootstrap.cc:492] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap complete.
I20250814 07:58:53.711939 25310 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 07:58:53.712563 25372 ts_tablet_manager.cc:1397] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Time spent bootstrapping tablet: real 0.185s user 0.147s sys 0.031s
I20250814 07:58:53.713397 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 25310
I20250814 07:58:53.739755 25372 raft_consensus.cc:357] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:53.743019 25372 raft_consensus.cc:738] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b744475f7264e8b82e4fd5b665570dc, State: Initialized, Role: FOLLOWER
I20250814 07:58:53.743825 25372 consensus_queue.cc:260] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } } peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } attrs { promote: false } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } }
I20250814 07:58:53.747918 25372 ts_tablet_manager.cc:1428] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Time spent starting tablet: real 0.035s user 0.029s sys 0.003s
I20250814 07:58:53.748495 25372 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap starting.
I20250814 07:58:53.756369 25480 heartbeater.cc:344] Connected to a master server at 127.20.120.62:40161
I20250814 07:58:53.756748 25480 heartbeater.cc:461] Registering TS with master...
I20250814 07:58:53.758002 25480 heartbeater.cc:507] Master 127.20.120.62:40161 requested a full tablet report, sending...
I20250814 07:58:53.761755 24969 ts_manager.cc:194] Registered new tserver with Master: 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755)
I20250814 07:58:53.764762 24969 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.3:35267
I20250814 07:58:53.769093 25480 heartbeater.cc:499] Master 127.20.120.62:40161 was elected leader, sending a full tablet report...
I20250814 07:58:53.769845 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:58:53.773612 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20250814 07:58:53.776825 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250814 07:58:53.798439 25415 tablet_service.cc:1515] Processing DeleteTablet for tablet a34062205d144210a396ae4b41bd9782 with delete_type TABLET_DATA_TOMBSTONED (TS 7b744475f7264e8b82e4fd5b665570dc not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:33088
I20250814 07:58:53.801203 25489 tablet_replica.cc:331] stopping tablet replica
I20250814 07:58:53.801724 25489 raft_consensus.cc:2241] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 FOLLOWER]: Raft consensus shutting down.
I20250814 07:58:53.802065 25489 raft_consensus.cc:2270] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc [term 2 FOLLOWER]: Raft consensus is shut down!
I20250814 07:58:53.804006 25489 ts_tablet_manager.cc:1905] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250814 07:58:53.813813 25489 ts_tablet_manager.cc:1918] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.11
I20250814 07:58:53.814051 25489 log.cc:1199] T a34062205d144210a396ae4b41bd9782 P 7b744475f7264e8b82e4fd5b665570dc: Deleting WAL directory at /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/wals/a34062205d144210a396ae4b41bd9782
I20250814 07:58:53.815179 24956 catalog_manager.cc:4928] TS 7b744475f7264e8b82e4fd5b665570dc (127.20.120.3:39755): tablet a34062205d144210a396ae4b41bd9782 (table TestTable [id=7daa466fbb1649fc932a146e8692db4d]) successfully deleted
I20250814 07:58:53.846082 25372 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 07:58:53.846670 25372 tablet_bootstrap.cc:492] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Bootstrap complete.
I20250814 07:58:53.847723 25372 ts_tablet_manager.cc:1397] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Time spent bootstrapping tablet: real 0.099s user 0.087s sys 0.008s
I20250814 07:58:53.849246 25372 raft_consensus.cc:357] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
I20250814 07:58:53.849665 25372 raft_consensus.cc:738] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b744475f7264e8b82e4fd5b665570dc, State: Initialized, Role: FOLLOWER
I20250814 07:58:53.850455 25372 consensus_queue.cc:260] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "73230c32551f4370926dd72ed159266e" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 46459 } } peers { permanent_uuid: "38b524e02a3b42ee8cfc21431efded38" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 46099 } attrs { promote: false } } peers { permanent_uuid: "7b744475f7264e8b82e4fd5b665570dc" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 39755 } attrs { promote: false } }
I20250814 07:58:53.851949 25372 ts_tablet_manager.cc:1428] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250814 07:58:54.264503 25435 raft_consensus.cc:3058] T 967c771199494673af94bd0b45841b7d P 7b744475f7264e8b82e4fd5b665570dc [term 2 FOLLOWER]: Advancing to term 3
W20250814 07:58:54.780233 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:58:55.783778 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:58:56.786960 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:58:57.790268 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:58:58.793315 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:58:59.796732 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:00.800649 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:01.804168 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:02.807330 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:03.810361 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:04.813582 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:05.817186 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:06.820420 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:07.823376 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:08.826547 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:09.829499 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:10.833174 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:11.836777 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250814 07:59:12.839972 20960 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet a34062205d144210a396ae4b41bd9782: tablet_id: "a34062205d144210a396ae4b41bd9782" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250814 07:59:13.842784 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 25007
I20250814 07:59:13.869700 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 25163
I20250814 07:59:13.894845 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 25310
I20250814 07:59:13.920411 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 24937
2025-08-14T07:59:13Z chronyd exiting
I20250814 07:59:13.966434 20960 test_util.cc:183] -----------------------------------------------
I20250814 07:59:13.966615 20960 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1755158230225856-20960-0
[ FAILED ] AdminCliTest.TestRebuildTables (58208 ms)
[----------] 5 tests from AdminCliTest (123676 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250814 07:59:13.970392 20960 test_util.cc:276] Using random seed: -1506376509
I20250814 07:59:13.974282 20960 ts_itest-base.cc:115] Starting cluster with:
I20250814 07:59:13.974434 20960 ts_itest-base.cc:116] --------------
I20250814 07:59:13.974578 20960 ts_itest-base.cc:117] 5 tablet servers
I20250814 07:59:13.974711 20960 ts_itest-base.cc:118] 3 replicas per TS
I20250814 07:59:13.974841 20960 ts_itest-base.cc:119] --------------
2025-08-14T07:59:13Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T07:59:13Z Disabled control of system clock
I20250814 07:59:14.013252 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:41321
--webserver_interface=127.20.120.62
--webserver_port=0
--builtin_ntp_servers=127.20.120.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:41321
--raft_prepare_replacement_before_eviction=true with env {}
W20250814 07:59:14.302371 25514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:14.302870 25514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:14.303252 25514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:14.331377 25514 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 07:59:14.331687 25514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:59:14.331888 25514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:14.332077 25514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:59:14.332262 25514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:59:14.364362 25514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:41321
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:41321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:14.365484 25514 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:14.366935 25514 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:14.378037 25520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:15.781103 25519 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 25514
W20250814 07:59:16.112154 25519 kernel_stack_watchdog.cc:198] Thread 25514 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 07:59:14.378412 25521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:16.112946 25514 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.735s user 0.000s sys 0.002s
W20250814 07:59:16.113305 25514 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.736s user 0.001s sys 0.002s
W20250814 07:59:16.113577 25522 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1734 milliseconds
I20250814 07:59:16.114868 25514 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250814 07:59:16.114956 25523 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:59:16.117830 25514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:16.120162 25514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:16.121506 25514 hybrid_clock.cc:648] HybridClock initialized: now 1755158356121461 us; error 50 us; skew 500 ppm
I20250814 07:59:16.122289 25514 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:16.128120 25514 webserver.cc:480] Webserver started at http://127.20.120.62:36183/ using document root <none> and password file <none>
I20250814 07:59:16.128976 25514 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:16.129207 25514 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:16.129634 25514 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:16.133980 25514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "395fc38084af46b0a57c891e26f7fa96"
format_stamp: "Formatted at 2025-08-14 07:59:16 on dist-test-slave-87c1"
I20250814 07:59:16.135010 25514 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "395fc38084af46b0a57c891e26f7fa96"
format_stamp: "Formatted at 2025-08-14 07:59:16 on dist-test-slave-87c1"
I20250814 07:59:16.142006 25514 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250814 07:59:16.147226 25530 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:16.148169 25514 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250814 07:59:16.148483 25514 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "395fc38084af46b0a57c891e26f7fa96"
format_stamp: "Formatted at 2025-08-14 07:59:16 on dist-test-slave-87c1"
I20250814 07:59:16.148795 25514 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:16.196821 25514 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:16.198223 25514 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:16.198678 25514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:16.266018 25514 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:41321
I20250814 07:59:16.266057 25581 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:41321 every 8 connection(s)
I20250814 07:59:16.268633 25514 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 07:59:16.273478 25582 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:16.276401 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 25514
I20250814 07:59:16.276806 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 07:59:16.295684 25582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96: Bootstrap starting.
I20250814 07:59:16.301223 25582 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:16.303232 25582 log.cc:826] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:16.308028 25582 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96: No bootstrap required, opened a new log
I20250814 07:59:16.324710 25582 raft_consensus.cc:357] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395fc38084af46b0a57c891e26f7fa96" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41321 } }
I20250814 07:59:16.325307 25582 raft_consensus.cc:383] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:16.325497 25582 raft_consensus.cc:738] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 395fc38084af46b0a57c891e26f7fa96, State: Initialized, Role: FOLLOWER
I20250814 07:59:16.326076 25582 consensus_queue.cc:260] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395fc38084af46b0a57c891e26f7fa96" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41321 } }
I20250814 07:59:16.326534 25582 raft_consensus.cc:397] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:59:16.326756 25582 raft_consensus.cc:491] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:59:16.327006 25582 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:16.330875 25582 raft_consensus.cc:513] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395fc38084af46b0a57c891e26f7fa96" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41321 } }
I20250814 07:59:16.331537 25582 leader_election.cc:304] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 395fc38084af46b0a57c891e26f7fa96; no voters:
I20250814 07:59:16.333204 25582 leader_election.cc:290] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:59:16.333853 25587 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:59:16.335994 25587 raft_consensus.cc:695] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [term 1 LEADER]: Becoming Leader. State: Replica: 395fc38084af46b0a57c891e26f7fa96, State: Running, Role: LEADER
I20250814 07:59:16.336751 25587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395fc38084af46b0a57c891e26f7fa96" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41321 } }
I20250814 07:59:16.337807 25582 sys_catalog.cc:564] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:59:16.346563 25588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "395fc38084af46b0a57c891e26f7fa96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395fc38084af46b0a57c891e26f7fa96" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41321 } } }
I20250814 07:59:16.347537 25588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [sys.catalog]: This master's current role is: LEADER
I20250814 07:59:16.348398 25589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 395fc38084af46b0a57c891e26f7fa96. Latest consensus state: current_term: 1 leader_uuid: "395fc38084af46b0a57c891e26f7fa96" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "395fc38084af46b0a57c891e26f7fa96" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 41321 } } }
I20250814 07:59:16.349397 25589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96 [sys.catalog]: This master's current role is: LEADER
I20250814 07:59:16.351409 25596 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:59:16.363453 25596 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:59:16.378480 25596 catalog_manager.cc:1349] Generated new cluster ID: bf54c7f4758349bc9127cac7f982372f
I20250814 07:59:16.378746 25596 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:59:16.407804 25596 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 07:59:16.409714 25596 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:59:16.429059 25596 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 395fc38084af46b0a57c891e26f7fa96: Generated new TSK 0
I20250814 07:59:16.429788 25596 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 07:59:16.445439 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:0
--local_ip_for_outbound_sockets=127.20.120.1
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41321
--builtin_ntp_servers=127.20.120.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250814 07:59:16.736133 25606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:16.736600 25606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:16.737177 25606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:16.767096 25606 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 07:59:16.767484 25606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:16.768221 25606 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 07:59:16.799736 25606 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41321
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:16.800936 25606 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:16.802630 25606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:16.814191 25612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:16.815310 25613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:18.217342 25611 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 25606
W20250814 07:59:18.218636 25606 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.405s user 0.480s sys 0.925s
W20250814 07:59:18.221448 25606 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.408s user 0.480s sys 0.925s
I20250814 07:59:18.226312 25606 server_base.cc:1047] running on GCE node
W20250814 07:59:18.227921 25617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:59:18.228986 25606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:18.231073 25606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:18.232401 25606 hybrid_clock.cc:648] HybridClock initialized: now 1755158358232382 us; error 39 us; skew 500 ppm
I20250814 07:59:18.233256 25606 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:18.239593 25606 webserver.cc:480] Webserver started at http://127.20.120.1:33603/ using document root <none> and password file <none>
I20250814 07:59:18.240463 25606 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:18.240689 25606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:18.241168 25606 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:18.245308 25606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "06bbf90e1c314b4f985b0b17f06bbaaf"
format_stamp: "Formatted at 2025-08-14 07:59:18 on dist-test-slave-87c1"
I20250814 07:59:18.246356 25606 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "06bbf90e1c314b4f985b0b17f06bbaaf"
format_stamp: "Formatted at 2025-08-14 07:59:18 on dist-test-slave-87c1"
I20250814 07:59:18.253832 25606 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250814 07:59:18.260843 25622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:18.261917 25606 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250814 07:59:18.262284 25606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "06bbf90e1c314b4f985b0b17f06bbaaf"
format_stamp: "Formatted at 2025-08-14 07:59:18 on dist-test-slave-87c1"
I20250814 07:59:18.262686 25606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:18.349171 25606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:18.351054 25606 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:18.351586 25606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:18.354732 25606 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:59:18.360081 25606 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:59:18.360357 25606 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:18.360647 25606 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:59:18.360850 25606 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:18.508566 25606 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:45469
I20250814 07:59:18.508749 25734 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:45469 every 8 connection(s)
I20250814 07:59:18.511111 25606 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 07:59:18.512122 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 25606
I20250814 07:59:18.512662 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 07:59:18.522872 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.2:0
--local_ip_for_outbound_sockets=127.20.120.2
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41321
--builtin_ntp_servers=127.20.120.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250814 07:59:18.540306 25735 heartbeater.cc:344] Connected to a master server at 127.20.120.62:41321
I20250814 07:59:18.540823 25735 heartbeater.cc:461] Registering TS with master...
I20250814 07:59:18.542177 25735 heartbeater.cc:507] Master 127.20.120.62:41321 requested a full tablet report, sending...
I20250814 07:59:18.545310 25547 ts_manager.cc:194] Registered new tserver with Master: 06bbf90e1c314b4f985b0b17f06bbaaf (127.20.120.1:45469)
I20250814 07:59:18.548007 25547 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:38685
W20250814 07:59:18.832695 25739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:18.833169 25739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:18.833683 25739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:18.863099 25739 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 07:59:18.863457 25739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:18.864187 25739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.2
I20250814 07:59:18.896411 25739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41321
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.2
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:18.897614 25739 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:18.899261 25739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:18.913914 25745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:59:19.552290 25735 heartbeater.cc:499] Master 127.20.120.62:41321 was elected leader, sending a full tablet report...
W20250814 07:59:18.914785 25746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:20.046608 25747 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250814 07:59:20.047641 25748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:59:20.047636 25739 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:59:20.050792 25739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:20.052928 25739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:20.054333 25739 hybrid_clock.cc:648] HybridClock initialized: now 1755158360054302 us; error 37 us; skew 500 ppm
I20250814 07:59:20.055121 25739 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:20.060979 25739 webserver.cc:480] Webserver started at http://127.20.120.2:40027/ using document root <none> and password file <none>
I20250814 07:59:20.061892 25739 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:20.062108 25739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:20.062574 25739 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:20.067229 25739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "cfbdde4746ba4837b10b02d3bb52f914"
format_stamp: "Formatted at 2025-08-14 07:59:20 on dist-test-slave-87c1"
I20250814 07:59:20.068277 25739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "cfbdde4746ba4837b10b02d3bb52f914"
format_stamp: "Formatted at 2025-08-14 07:59:20 on dist-test-slave-87c1"
I20250814 07:59:20.075003 25739 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20250814 07:59:20.080094 25755 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:20.081202 25739 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250814 07:59:20.081514 25739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "cfbdde4746ba4837b10b02d3bb52f914"
format_stamp: "Formatted at 2025-08-14 07:59:20 on dist-test-slave-87c1"
I20250814 07:59:20.081821 25739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:20.141278 25739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:20.142566 25739 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:20.142962 25739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:20.145222 25739 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:59:20.148788 25739 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:59:20.148991 25739 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:20.149237 25739 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:59:20.149389 25739 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:20.284436 25739 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.2:35383
I20250814 07:59:20.284535 25867 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.2:35383 every 8 connection(s)
I20250814 07:59:20.286857 25739 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250814 07:59:20.295209 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 25739
I20250814 07:59:20.295612 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250814 07:59:20.301769 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.3:0
--local_ip_for_outbound_sockets=127.20.120.3
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41321
--builtin_ntp_servers=127.20.120.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250814 07:59:20.306293 25868 heartbeater.cc:344] Connected to a master server at 127.20.120.62:41321
I20250814 07:59:20.306751 25868 heartbeater.cc:461] Registering TS with master...
I20250814 07:59:20.307938 25868 heartbeater.cc:507] Master 127.20.120.62:41321 requested a full tablet report, sending...
I20250814 07:59:20.310148 25547 ts_manager.cc:194] Registered new tserver with Master: cfbdde4746ba4837b10b02d3bb52f914 (127.20.120.2:35383)
I20250814 07:59:20.311416 25547 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.2:39709
W20250814 07:59:20.600106 25872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:20.600538 25872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:20.600951 25872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:20.629653 25872 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 07:59:20.629973 25872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:20.630653 25872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.3
I20250814 07:59:20.661564 25872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41321
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.3
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:20.662621 25872 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:20.664006 25872 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:20.674753 25879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:59:21.314301 25868 heartbeater.cc:499] Master 127.20.120.62:41321 was elected leader, sending a full tablet report...
W20250814 07:59:20.675758 25880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:21.852051 25882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:21.855171 25881 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1179 milliseconds
W20250814 07:59:21.856287 25872 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.181s user 0.447s sys 0.720s
W20250814 07:59:21.856539 25872 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.182s user 0.447s sys 0.720s
I20250814 07:59:21.856750 25872 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:59:21.857772 25872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:21.865653 25872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:21.867076 25872 hybrid_clock.cc:648] HybridClock initialized: now 1755158361867012 us; error 65 us; skew 500 ppm
I20250814 07:59:21.867949 25872 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:21.874697 25872 webserver.cc:480] Webserver started at http://127.20.120.3:35261/ using document root <none> and password file <none>
I20250814 07:59:21.875555 25872 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:21.875769 25872 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:21.876196 25872 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:21.880345 25872 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "0d2242a58d1040af8b2c6e46ae64c150"
format_stamp: "Formatted at 2025-08-14 07:59:21 on dist-test-slave-87c1"
I20250814 07:59:21.881587 25872 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "0d2242a58d1040af8b2c6e46ae64c150"
format_stamp: "Formatted at 2025-08-14 07:59:21 on dist-test-slave-87c1"
I20250814 07:59:21.888625 25872 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20250814 07:59:21.894165 25889 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:21.895241 25872 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250814 07:59:21.895552 25872 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "0d2242a58d1040af8b2c6e46ae64c150"
format_stamp: "Formatted at 2025-08-14 07:59:21 on dist-test-slave-87c1"
I20250814 07:59:21.895859 25872 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:21.964217 25872 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:21.965544 25872 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:21.965957 25872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:21.968266 25872 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:59:21.972150 25872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:59:21.972337 25872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:21.972584 25872 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:59:21.972741 25872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:22.101034 25872 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.3:44511
I20250814 07:59:22.101159 26001 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.3:44511 every 8 connection(s)
I20250814 07:59:22.103534 25872 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250814 07:59:22.113602 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 25872
I20250814 07:59:22.114151 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250814 07:59:22.121086 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.4:0
--local_ip_for_outbound_sockets=127.20.120.4
--webserver_interface=127.20.120.4
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41321
--builtin_ntp_servers=127.20.120.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250814 07:59:22.126169 26002 heartbeater.cc:344] Connected to a master server at 127.20.120.62:41321
I20250814 07:59:22.126679 26002 heartbeater.cc:461] Registering TS with master...
I20250814 07:59:22.127914 26002 heartbeater.cc:507] Master 127.20.120.62:41321 requested a full tablet report, sending...
I20250814 07:59:22.130163 25547 ts_manager.cc:194] Registered new tserver with Master: 0d2242a58d1040af8b2c6e46ae64c150 (127.20.120.3:44511)
I20250814 07:59:22.131853 25547 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.3:46031
W20250814 07:59:22.411586 26006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:22.412012 26006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:22.412456 26006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:22.441807 26006 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 07:59:22.442111 26006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:22.442857 26006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.4
I20250814 07:59:22.475106 26006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.4:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.20.120.4
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41321
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.4
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:22.476356 26006 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:22.477825 26006 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:22.489545 26012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:59:23.134872 26002 heartbeater.cc:499] Master 127.20.120.62:41321 was elected leader, sending a full tablet report...
W20250814 07:59:22.490326 26013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:23.676889 26015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:23.679633 26014 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1183 milliseconds
W20250814 07:59:23.680402 26006 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.191s user 0.397s sys 0.785s
W20250814 07:59:23.680672 26006 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.191s user 0.397s sys 0.785s
I20250814 07:59:23.680864 26006 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:59:23.681916 26006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:23.684062 26006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:23.685386 26006 hybrid_clock.cc:648] HybridClock initialized: now 1755158363685356 us; error 34 us; skew 500 ppm
I20250814 07:59:23.686142 26006 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:23.693163 26006 webserver.cc:480] Webserver started at http://127.20.120.4:46377/ using document root <none> and password file <none>
I20250814 07:59:23.694109 26006 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:23.694316 26006 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:23.694741 26006 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:23.698889 26006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "37d3e2b105af454297577be29212ef79"
format_stamp: "Formatted at 2025-08-14 07:59:23 on dist-test-slave-87c1"
I20250814 07:59:23.699889 26006 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "37d3e2b105af454297577be29212ef79"
format_stamp: "Formatted at 2025-08-14 07:59:23 on dist-test-slave-87c1"
I20250814 07:59:23.707134 26006 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250814 07:59:23.712532 26022 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:23.713531 26006 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250814 07:59:23.713799 26006 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "37d3e2b105af454297577be29212ef79"
format_stamp: "Formatted at 2025-08-14 07:59:23 on dist-test-slave-87c1"
I20250814 07:59:23.714078 26006 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:23.785956 26006 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:23.787423 26006 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:23.787858 26006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:23.790359 26006 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:59:23.794409 26006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:59:23.794610 26006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:23.794862 26006 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:59:23.795017 26006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:23.923507 26006 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.4:39407
I20250814 07:59:23.923597 26134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.4:39407 every 8 connection(s)
I20250814 07:59:23.925868 26006 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250814 07:59:23.936456 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 26006
I20250814 07:59:23.936949 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250814 07:59:23.943110 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.5:0
--local_ip_for_outbound_sockets=127.20.120.5
--webserver_interface=127.20.120.5
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41321
--builtin_ntp_servers=127.20.120.20:39309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250814 07:59:23.945173 26135 heartbeater.cc:344] Connected to a master server at 127.20.120.62:41321
I20250814 07:59:23.945683 26135 heartbeater.cc:461] Registering TS with master...
I20250814 07:59:23.946915 26135 heartbeater.cc:507] Master 127.20.120.62:41321 requested a full tablet report, sending...
I20250814 07:59:23.949183 25547 ts_manager.cc:194] Registered new tserver with Master: 37d3e2b105af454297577be29212ef79 (127.20.120.4:39407)
I20250814 07:59:23.950361 25547 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.4:58059
W20250814 07:59:24.227087 26139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:24.227535 26139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:24.228018 26139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:24.256474 26139 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250814 07:59:24.256844 26139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:24.257617 26139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.5
I20250814 07:59:24.296774 26139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:39309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.5:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.20.120.5
--webserver_port=0
--tserver_master_addrs=127.20.120.62:41321
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.5
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:24.298162 26139 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:24.299623 26139 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:24.311069 26146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:59:24.953510 26135 heartbeater.cc:499] Master 127.20.120.62:41321 was elected leader, sending a full tablet report...
W20250814 07:59:24.311921 26147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:25.650094 26149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:25.654548 26148 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1341 milliseconds
W20250814 07:59:25.655045 26139 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.343s user 0.478s sys 0.850s
W20250814 07:59:25.655311 26139 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.344s user 0.478s sys 0.850s
I20250814 07:59:25.655512 26139 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:59:25.656492 26139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:25.658679 26139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:25.660018 26139 hybrid_clock.cc:648] HybridClock initialized: now 1755158365659987 us; error 37 us; skew 500 ppm
I20250814 07:59:25.660805 26139 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:25.668391 26139 webserver.cc:480] Webserver started at http://127.20.120.5:46309/ using document root <none> and password file <none>
I20250814 07:59:25.669378 26139 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:25.669610 26139 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:25.670032 26139 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:25.674378 26139 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e6ffa2bb960d4647a62b7c73812d352e"
format_stamp: "Formatted at 2025-08-14 07:59:25 on dist-test-slave-87c1"
I20250814 07:59:25.675431 26139 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e6ffa2bb960d4647a62b7c73812d352e"
format_stamp: "Formatted at 2025-08-14 07:59:25 on dist-test-slave-87c1"
I20250814 07:59:25.683086 26139 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.006s
I20250814 07:59:25.688700 26156 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:25.689765 26139 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20250814 07:59:25.690047 26139 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e6ffa2bb960d4647a62b7c73812d352e"
format_stamp: "Formatted at 2025-08-14 07:59:25 on dist-test-slave-87c1"
I20250814 07:59:25.690349 26139 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:25.750000 26139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:25.751315 26139 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:25.751724 26139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:25.754091 26139 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:59:25.757891 26139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:59:25.758096 26139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:25.758342 26139 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:59:25.758507 26139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:25.888598 26139 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.5:46295
I20250814 07:59:25.888696 26268 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.5:46295 every 8 connection(s)
I20250814 07:59:25.890985 26139 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250814 07:59:25.900269 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 26139
I20250814 07:59:25.900640 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250814 07:59:25.910326 26269 heartbeater.cc:344] Connected to a master server at 127.20.120.62:41321
I20250814 07:59:25.910727 26269 heartbeater.cc:461] Registering TS with master...
I20250814 07:59:25.911718 26269 heartbeater.cc:507] Master 127.20.120.62:41321 requested a full tablet report, sending...
I20250814 07:59:25.913676 25547 ts_manager.cc:194] Registered new tserver with Master: e6ffa2bb960d4647a62b7c73812d352e (127.20.120.5:46295)
I20250814 07:59:25.914906 25547 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.5:43951
I20250814 07:59:25.920233 20960 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250814 07:59:25.953281 25547 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37140:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250814 07:59:26.037099 25937 tablet_service.cc:1468] Processing CreateTablet for tablet 7985b44fa3b643c19315fe13c54fe718 (DEFAULT_TABLE table=TestTable [id=41996c5838b64b23b34d80342957140b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:59:26.038599 25937 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7985b44fa3b643c19315fe13c54fe718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:26.037909 26204 tablet_service.cc:1468] Processing CreateTablet for tablet 7985b44fa3b643c19315fe13c54fe718 (DEFAULT_TABLE table=TestTable [id=41996c5838b64b23b34d80342957140b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:59:26.038235 25670 tablet_service.cc:1468] Processing CreateTablet for tablet 7985b44fa3b643c19315fe13c54fe718 (DEFAULT_TABLE table=TestTable [id=41996c5838b64b23b34d80342957140b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:59:26.039780 26204 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7985b44fa3b643c19315fe13c54fe718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:26.040035 25670 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7985b44fa3b643c19315fe13c54fe718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:26.059489 26288 tablet_bootstrap.cc:492] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150: Bootstrap starting.
I20250814 07:59:26.065308 26288 tablet_bootstrap.cc:654] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:26.068028 26289 tablet_bootstrap.cc:492] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf: Bootstrap starting.
I20250814 07:59:26.067744 26288 log.cc:826] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:26.069568 26290 tablet_bootstrap.cc:492] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e: Bootstrap starting.
I20250814 07:59:26.074963 26288 tablet_bootstrap.cc:492] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150: No bootstrap required, opened a new log
I20250814 07:59:26.075414 26289 tablet_bootstrap.cc:654] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:26.075470 26288 ts_tablet_manager.cc:1397] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20250814 07:59:26.075616 26290 tablet_bootstrap.cc:654] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:26.077311 26290 log.cc:826] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:26.077720 26289 log.cc:826] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:26.082944 26290 tablet_bootstrap.cc:492] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e: No bootstrap required, opened a new log
I20250814 07:59:26.083405 26290 ts_tablet_manager.cc:1397] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20250814 07:59:26.084084 26289 tablet_bootstrap.cc:492] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf: No bootstrap required, opened a new log
I20250814 07:59:26.084455 26289 ts_tablet_manager.cc:1397] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf: Time spent bootstrapping tablet: real 0.017s user 0.007s sys 0.007s
I20250814 07:59:26.103279 26289 raft_consensus.cc:357] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 } } peers { permanent_uuid: "0d2242a58d1040af8b2c6e46ae64c150" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44511 } } peers { permanent_uuid: "e6ffa2bb960d4647a62b7c73812d352e" member_type: VOTER last_known_addr { host: "127.20.120.5" port: 46295 } }
I20250814 07:59:26.104192 26289 raft_consensus.cc:383] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:26.104521 26289 raft_consensus.cc:738] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06bbf90e1c314b4f985b0b17f06bbaaf, State: Initialized, Role: FOLLOWER
I20250814 07:59:26.104234 26288 raft_consensus.cc:357] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 } } peers { permanent_uuid: "0d2242a58d1040af8b2c6e46ae64c150" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44511 } } peers { permanent_uuid: "e6ffa2bb960d4647a62b7c73812d352e" member_type: VOTER last_known_addr { host: "127.20.120.5" port: 46295 } }
I20250814 07:59:26.105187 26288 raft_consensus.cc:383] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:26.105533 26288 raft_consensus.cc:738] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d2242a58d1040af8b2c6e46ae64c150, State: Initialized, Role: FOLLOWER
I20250814 07:59:26.105473 26289 consensus_queue.cc:260] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 } } peers { permanent_uuid: "0d2242a58d1040af8b2c6e46ae64c150" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44511 } } peers { permanent_uuid: "e6ffa2bb960d4647a62b7c73812d352e" member_type: VOTER last_known_addr { host: "127.20.120.5" port: 46295 } }
I20250814 07:59:26.106484 26288 consensus_queue.cc:260] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 } } peers { permanent_uuid: "0d2242a58d1040af8b2c6e46ae64c150" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44511 } } peers { permanent_uuid: "e6ffa2bb960d4647a62b7c73812d352e" member_type: VOTER last_known_addr { host: "127.20.120.5" port: 46295 } }
I20250814 07:59:26.110080 26290 raft_consensus.cc:357] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 } } peers { permanent_uuid: "0d2242a58d1040af8b2c6e46ae64c150" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44511 } } peers { permanent_uuid: "e6ffa2bb960d4647a62b7c73812d352e" member_type: VOTER last_known_addr { host: "127.20.120.5" port: 46295 } }
I20250814 07:59:26.111251 26290 raft_consensus.cc:383] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:26.111685 26290 raft_consensus.cc:738] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ffa2bb960d4647a62b7c73812d352e, State: Initialized, Role: FOLLOWER
I20250814 07:59:26.112910 26290 consensus_queue.cc:260] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 } } peers { permanent_uuid: "0d2242a58d1040af8b2c6e46ae64c150" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44511 } } peers { permanent_uuid: "e6ffa2bb960d4647a62b7c73812d352e" member_type: VOTER last_known_addr { host: "127.20.120.5" port: 46295 } }
I20250814 07:59:26.116235 26288 ts_tablet_manager.cc:1428] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150: Time spent starting tablet: real 0.041s user 0.036s sys 0.004s
I20250814 07:59:26.127323 26289 ts_tablet_manager.cc:1428] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf: Time spent starting tablet: real 0.043s user 0.022s sys 0.019s
I20250814 07:59:26.131704 26269 heartbeater.cc:499] Master 127.20.120.62:41321 was elected leader, sending a full tablet report...
I20250814 07:59:26.132931 26290 ts_tablet_manager.cc:1428] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e: Time spent starting tablet: real 0.049s user 0.028s sys 0.018s
W20250814 07:59:26.145364 26270 tablet.cc:2378] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 07:59:26.277309 25736 tablet.cc:2378] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:59:26.309720 26295 raft_consensus.cc:491] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:59:26.310174 26295 raft_consensus.cc:513] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 } } peers { permanent_uuid: "0d2242a58d1040af8b2c6e46ae64c150" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44511 } } peers { permanent_uuid: "e6ffa2bb960d4647a62b7c73812d352e" member_type: VOTER last_known_addr { host: "127.20.120.5" port: 46295 } }
I20250814 07:59:26.312626 26295 leader_election.cc:290] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06bbf90e1c314b4f985b0b17f06bbaaf (127.20.120.1:45469), e6ffa2bb960d4647a62b7c73812d352e (127.20.120.5:46295)
I20250814 07:59:26.326161 26224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7985b44fa3b643c19315fe13c54fe718" candidate_uuid: "0d2242a58d1040af8b2c6e46ae64c150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ffa2bb960d4647a62b7c73812d352e" is_pre_election: true
I20250814 07:59:26.326159 25690 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7985b44fa3b643c19315fe13c54fe718" candidate_uuid: "0d2242a58d1040af8b2c6e46ae64c150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" is_pre_election: true
I20250814 07:59:26.327005 26224 raft_consensus.cc:2466] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d2242a58d1040af8b2c6e46ae64c150 in term 0.
I20250814 07:59:26.327005 25690 raft_consensus.cc:2466] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d2242a58d1040af8b2c6e46ae64c150 in term 0.
I20250814 07:59:26.328210 25891 leader_election.cc:304] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06bbf90e1c314b4f985b0b17f06bbaaf, 0d2242a58d1040af8b2c6e46ae64c150; no voters:
I20250814 07:59:26.328847 26295 raft_consensus.cc:2802] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 07:59:26.329209 26295 raft_consensus.cc:491] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 07:59:26.329427 26295 raft_consensus.cc:3058] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:26.333947 26295 raft_consensus.cc:513] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 } } peers { permanent_uuid: "0d2242a58d1040af8b2c6e46ae64c150" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44511 } } peers { permanent_uuid: "e6ffa2bb960d4647a62b7c73812d352e" member_type: VOTER last_known_addr { host: "127.20.120.5" port: 46295 } }
I20250814 07:59:26.335314 26295 leader_election.cc:290] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [CANDIDATE]: Term 1 election: Requested vote from peers 06bbf90e1c314b4f985b0b17f06bbaaf (127.20.120.1:45469), e6ffa2bb960d4647a62b7c73812d352e (127.20.120.5:46295)
I20250814 07:59:26.336035 25690 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7985b44fa3b643c19315fe13c54fe718" candidate_uuid: "0d2242a58d1040af8b2c6e46ae64c150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf"
I20250814 07:59:26.336287 26224 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7985b44fa3b643c19315fe13c54fe718" candidate_uuid: "0d2242a58d1040af8b2c6e46ae64c150" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ffa2bb960d4647a62b7c73812d352e"
I20250814 07:59:26.336452 25690 raft_consensus.cc:3058] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:26.336687 26224 raft_consensus.cc:3058] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:26.341248 25690 raft_consensus.cc:2466] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d2242a58d1040af8b2c6e46ae64c150 in term 1.
I20250814 07:59:26.341815 26224 raft_consensus.cc:2466] T 7985b44fa3b643c19315fe13c54fe718 P e6ffa2bb960d4647a62b7c73812d352e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d2242a58d1040af8b2c6e46ae64c150 in term 1.
I20250814 07:59:26.342038 25891 leader_election.cc:304] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06bbf90e1c314b4f985b0b17f06bbaaf, 0d2242a58d1040af8b2c6e46ae64c150; no voters:
I20250814 07:59:26.342643 26295 raft_consensus.cc:2802] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:59:26.344219 26295 raft_consensus.cc:695] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [term 1 LEADER]: Becoming Leader. State: Replica: 0d2242a58d1040af8b2c6e46ae64c150, State: Running, Role: LEADER
I20250814 07:59:26.345084 26295 consensus_queue.cc:237] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 } } peers { permanent_uuid: "0d2242a58d1040af8b2c6e46ae64c150" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44511 } } peers { permanent_uuid: "e6ffa2bb960d4647a62b7c73812d352e" member_type: VOTER last_known_addr { host: "127.20.120.5" port: 46295 } }
I20250814 07:59:26.353703 25547 catalog_manager.cc:5582] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d2242a58d1040af8b2c6e46ae64c150 (127.20.120.3). New cstate: current_term: 1 leader_uuid: "0d2242a58d1040af8b2c6e46ae64c150" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d2242a58d1040af8b2c6e46ae64c150" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e6ffa2bb960d4647a62b7c73812d352e" member_type: VOTER last_known_addr { host: "127.20.120.5" port: 46295 } health_report { overall_health: UNKNOWN } } }
W20250814 07:59:26.362048 26003 tablet.cc:2378] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:59:26.396018 20960 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250814 07:59:26.399189 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 06bbf90e1c314b4f985b0b17f06bbaaf to finish bootstrapping
I20250814 07:59:26.410856 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 0d2242a58d1040af8b2c6e46ae64c150 to finish bootstrapping
I20250814 07:59:26.420111 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver e6ffa2bb960d4647a62b7c73812d352e to finish bootstrapping
I20250814 07:59:26.429917 20960 test_util.cc:276] Using random seed: -1493916982
I20250814 07:59:26.452669 20960 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250814 07:59:26.453344 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 26139
W20250814 07:59:26.484236 25893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.5:46295: connect: Connection refused (error 111)
I20250814 07:59:26.484632 25690 raft_consensus.cc:1273] T 7985b44fa3b643c19315fe13c54fe718 P 06bbf90e1c314b4f985b0b17f06bbaaf [term 1 FOLLOWER]: Refusing update from remote peer 0d2242a58d1040af8b2c6e46ae64c150: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250814 07:59:26.486641 26299 consensus_queue.cc:1035] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06bbf90e1c314b4f985b0b17f06bbaaf" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 45469 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250814 07:59:26.489876 25893 consensus_peers.cc:489] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 -> Peer e6ffa2bb960d4647a62b7c73812d352e (127.20.120.5:46295): Couldn't send request to peer e6ffa2bb960d4647a62b7c73812d352e. Status: Network error: Client connection negotiation failed: client connection to 127.20.120.5:46295: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250814 07:59:26.505290 26309 mvcc.cc:204] Tried to move back new op lower bound from 7189128669103046656 to 7189128668555816960. Current Snapshot: MvccSnapshot[applied={T|T < 7189128669103046656}]
I20250814 07:59:26.508133 26312 mvcc.cc:204] Tried to move back new op lower bound from 7189128669103046656 to 7189128668555816960. Current Snapshot: MvccSnapshot[applied={T|T < 7189128669103046656}]
W20250814 07:59:29.040840 25893 consensus_peers.cc:489] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 -> Peer e6ffa2bb960d4647a62b7c73812d352e (127.20.120.5:46295): Couldn't send request to peer e6ffa2bb960d4647a62b7c73812d352e. Status: Network error: Client connection negotiation failed: client connection to 127.20.120.5:46295: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250814 07:59:29.132642 25670 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 07:59:29.163940 25937 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 07:59:29.166752 26070 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 07:59:29.225854 25803 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 07:59:30.924249 25670 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 07:59:30.955277 25937 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 07:59:30.985816 26070 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 07:59:31.018388 25803 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250814 07:59:31.428325 25893 consensus_peers.cc:489] T 7985b44fa3b643c19315fe13c54fe718 P 0d2242a58d1040af8b2c6e46ae64c150 -> Peer e6ffa2bb960d4647a62b7c73812d352e (127.20.120.5:46295): Couldn't send request to peer e6ffa2bb960d4647a62b7c73812d352e. Status: Network error: Client connection negotiation failed: client connection to 127.20.120.5:46295: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20250814 07:59:31.988277 25893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.20.120.5:46295: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250814 07:59:33.442948 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 25606
I20250814 07:59:33.475625 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 25739
I20250814 07:59:33.500151 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 25872
I20250814 07:59:33.537703 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 26006
I20250814 07:59:33.562292 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 25514
2025-08-14T07:59:33Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (19651 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (19651 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250814 07:59:33.622161 20960 test_util.cc:276] Using random seed: -1486724753
I20250814 07:59:33.626219 20960 ts_itest-base.cc:115] Starting cluster with:
I20250814 07:59:33.626385 20960 ts_itest-base.cc:116] --------------
I20250814 07:59:33.626539 20960 ts_itest-base.cc:117] 1 tablet servers
I20250814 07:59:33.626675 20960 ts_itest-base.cc:118] 1 replicas per TS
I20250814 07:59:33.626827 20960 ts_itest-base.cc:119] --------------
2025-08-14T07:59:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T07:59:33Z Disabled control of system clock
I20250814 07:59:33.667011 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:33975
--webserver_interface=127.20.120.62
--webserver_port=0
--builtin_ntp_servers=127.20.120.20:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:33975 with env {}
W20250814 07:59:33.959136 26406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:33.959774 26406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:33.960268 26406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:33.991432 26406 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:59:33.991816 26406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:33.992096 26406 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:59:33.992357 26406 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:59:34.026417 26406 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:33975
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:33975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:34.027711 26406 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:34.029307 26406 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:34.039058 26412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:35.443809 26411 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 26406
W20250814 07:59:34.040671 26413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:35.772920 26411 kernel_stack_watchdog.cc:198] Thread 26406 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 07:59:35.772985 26406 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.733s user 0.000s sys 0.002s
W20250814 07:59:35.773418 26406 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.733s user 0.000s sys 0.002s
W20250814 07:59:35.775116 26415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:35.777535 26414 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1731 milliseconds
I20250814 07:59:35.777558 26406 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:59:35.778764 26406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:35.781203 26406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:35.782541 26406 hybrid_clock.cc:648] HybridClock initialized: now 1755158375782491 us; error 48 us; skew 500 ppm
I20250814 07:59:35.783320 26406 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:35.789052 26406 webserver.cc:480] Webserver started at http://127.20.120.62:45679/ using document root <none> and password file <none>
I20250814 07:59:35.789932 26406 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:35.790138 26406 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:35.790573 26406 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:35.794718 26406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "becbf209e3204d1faffdec4909a808b9"
format_stamp: "Formatted at 2025-08-14 07:59:35 on dist-test-slave-87c1"
I20250814 07:59:35.795782 26406 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "becbf209e3204d1faffdec4909a808b9"
format_stamp: "Formatted at 2025-08-14 07:59:35 on dist-test-slave-87c1"
I20250814 07:59:35.802495 26406 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.004s
I20250814 07:59:35.807515 26422 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:35.808458 26406 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250814 07:59:35.808768 26406 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
uuid: "becbf209e3204d1faffdec4909a808b9"
format_stamp: "Formatted at 2025-08-14 07:59:35 on dist-test-slave-87c1"
I20250814 07:59:35.809108 26406 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:35.854653 26406 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:35.856009 26406 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:35.856467 26406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:35.924360 26406 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:33975
I20250814 07:59:35.924387 26473 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:33975 every 8 connection(s)
I20250814 07:59:35.927013 26406 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250814 07:59:35.931802 26474 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:35.933835 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 26406
I20250814 07:59:35.934357 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250814 07:59:35.952201 26474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9: Bootstrap starting.
I20250814 07:59:35.957793 26474 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:35.959666 26474 log.cc:826] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:35.963672 26474 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9: No bootstrap required, opened a new log
I20250814 07:59:35.979209 26474 raft_consensus.cc:357] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbf209e3204d1faffdec4909a808b9" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 33975 } }
I20250814 07:59:35.979748 26474 raft_consensus.cc:383] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:35.979972 26474 raft_consensus.cc:738] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: becbf209e3204d1faffdec4909a808b9, State: Initialized, Role: FOLLOWER
I20250814 07:59:35.980583 26474 consensus_queue.cc:260] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbf209e3204d1faffdec4909a808b9" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 33975 } }
I20250814 07:59:35.981047 26474 raft_consensus.cc:397] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:59:35.981282 26474 raft_consensus.cc:491] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:59:35.981655 26474 raft_consensus.cc:3058] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:35.985574 26474 raft_consensus.cc:513] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbf209e3204d1faffdec4909a808b9" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 33975 } }
I20250814 07:59:35.986194 26474 leader_election.cc:304] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: becbf209e3204d1faffdec4909a808b9; no voters:
I20250814 07:59:35.987748 26474 leader_election.cc:290] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:59:35.988456 26479 raft_consensus.cc:2802] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:59:35.990525 26479 raft_consensus.cc:695] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [term 1 LEADER]: Becoming Leader. State: Replica: becbf209e3204d1faffdec4909a808b9, State: Running, Role: LEADER
I20250814 07:59:35.991232 26479 consensus_queue.cc:237] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbf209e3204d1faffdec4909a808b9" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 33975 } }
I20250814 07:59:35.992128 26474 sys_catalog.cc:564] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:59:36.003280 26481 sys_catalog.cc:455] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader becbf209e3204d1faffdec4909a808b9. Latest consensus state: current_term: 1 leader_uuid: "becbf209e3204d1faffdec4909a808b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbf209e3204d1faffdec4909a808b9" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 33975 } } }
I20250814 07:59:36.003428 26480 sys_catalog.cc:455] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "becbf209e3204d1faffdec4909a808b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "becbf209e3204d1faffdec4909a808b9" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 33975 } } }
I20250814 07:59:36.003991 26481 sys_catalog.cc:458] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [sys.catalog]: This master's current role is: LEADER
I20250814 07:59:36.004004 26480 sys_catalog.cc:458] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9 [sys.catalog]: This master's current role is: LEADER
I20250814 07:59:36.006989 26488 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:59:36.018198 26488 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:59:36.034447 26488 catalog_manager.cc:1349] Generated new cluster ID: 08d15b0b77a34259bb1c2aa8e758a0ff
I20250814 07:59:36.034726 26488 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:59:36.069626 26488 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 07:59:36.070962 26488 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:59:36.083859 26488 catalog_manager.cc:5955] T 00000000000000000000000000000000 P becbf209e3204d1faffdec4909a808b9: Generated new TSK 0
I20250814 07:59:36.084830 26488 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 07:59:36.101590 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:0
--local_ip_for_outbound_sockets=127.20.120.1
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:33975
--builtin_ntp_servers=127.20.120.20:35049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250814 07:59:36.399564 26498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:36.400043 26498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:36.400548 26498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:36.430763 26498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:36.431589 26498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 07:59:36.465159 26498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:35049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:33975
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:36.466434 26498 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:36.467931 26498 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:36.480069 26504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:37.881176 26503 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 26498
W20250814 07:59:36.481639 26505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:38.196503 26503 kernel_stack_watchdog.cc:198] Thread 26498 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 07:59:38.202926 26506 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1721 milliseconds
W20250814 07:59:38.204126 26498 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.724s user 0.525s sys 1.154s
W20250814 07:59:38.204514 26507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:38.204561 26498 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.725s user 0.525s sys 1.154s
I20250814 07:59:38.205788 26498 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:59:38.207139 26498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:38.210180 26498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:38.211668 26498 hybrid_clock.cc:648] HybridClock initialized: now 1755158378211598 us; error 74 us; skew 500 ppm
I20250814 07:59:38.212678 26498 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:38.221287 26498 webserver.cc:480] Webserver started at http://127.20.120.1:39169/ using document root <none> and password file <none>
I20250814 07:59:38.222371 26498 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:38.222599 26498 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:38.223065 26498 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:38.228533 26498 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "6a7a2f82967646d2bc25bdf06c7a8ffd"
format_stamp: "Formatted at 2025-08-14 07:59:38 on dist-test-slave-87c1"
I20250814 07:59:38.229604 26498 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "6a7a2f82967646d2bc25bdf06c7a8ffd"
format_stamp: "Formatted at 2025-08-14 07:59:38 on dist-test-slave-87c1"
I20250814 07:59:38.236744 26498 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20250814 07:59:38.242259 26514 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:38.243242 26498 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250814 07:59:38.243521 26498 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "6a7a2f82967646d2bc25bdf06c7a8ffd"
format_stamp: "Formatted at 2025-08-14 07:59:38 on dist-test-slave-87c1"
I20250814 07:59:38.243793 26498 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:38.310129 26498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:38.311401 26498 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:38.311798 26498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:38.314561 26498 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:59:38.318810 26498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:59:38.318984 26498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:38.319248 26498 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:59:38.319387 26498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:38.486784 26498 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:40155
I20250814 07:59:38.486891 26626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:40155 every 8 connection(s)
I20250814 07:59:38.489420 26498 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250814 07:59:38.499349 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 26498
I20250814 07:59:38.499789 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1755158230225856-20960-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250814 07:59:38.511814 26627 heartbeater.cc:344] Connected to a master server at 127.20.120.62:33975
I20250814 07:59:38.512274 26627 heartbeater.cc:461] Registering TS with master...
I20250814 07:59:38.513466 26627 heartbeater.cc:507] Master 127.20.120.62:33975 requested a full tablet report, sending...
I20250814 07:59:38.515971 26439 ts_manager.cc:194] Registered new tserver with Master: 6a7a2f82967646d2bc25bdf06c7a8ffd (127.20.120.1:40155)
I20250814 07:59:38.518273 26439 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:59029
I20250814 07:59:38.519330 20960 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250814 07:59:38.552428 26439 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41498:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250814 07:59:38.610831 26562 tablet_service.cc:1468] Processing CreateTablet for tablet 76f6b56b1d2842338bda9f592f77a9f5 (DEFAULT_TABLE table=TestTable [id=fe0c300a63074226ae6c3bc531c7cd2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:59:38.612406 26562 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76f6b56b1d2842338bda9f592f77a9f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:38.633251 26642 tablet_bootstrap.cc:492] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd: Bootstrap starting.
I20250814 07:59:38.638916 26642 tablet_bootstrap.cc:654] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:38.640756 26642 log.cc:826] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:38.646373 26642 tablet_bootstrap.cc:492] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd: No bootstrap required, opened a new log
I20250814 07:59:38.646878 26642 ts_tablet_manager.cc:1397] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20250814 07:59:38.671319 26642 raft_consensus.cc:357] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7a2f82967646d2bc25bdf06c7a8ffd" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 40155 } }
I20250814 07:59:38.672057 26642 raft_consensus.cc:383] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:38.672374 26642 raft_consensus.cc:738] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a7a2f82967646d2bc25bdf06c7a8ffd, State: Initialized, Role: FOLLOWER
I20250814 07:59:38.673241 26642 consensus_queue.cc:260] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7a2f82967646d2bc25bdf06c7a8ffd" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 40155 } }
I20250814 07:59:38.673913 26642 raft_consensus.cc:397] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:59:38.674242 26642 raft_consensus.cc:491] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:59:38.674607 26642 raft_consensus.cc:3058] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:38.680847 26642 raft_consensus.cc:513] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7a2f82967646d2bc25bdf06c7a8ffd" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 40155 } }
I20250814 07:59:38.681712 26642 leader_election.cc:304] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a7a2f82967646d2bc25bdf06c7a8ffd; no voters:
I20250814 07:59:38.683770 26642 leader_election.cc:290] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:59:38.684095 26644 raft_consensus.cc:2802] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:59:38.690107 26627 heartbeater.cc:499] Master 127.20.120.62:33975 was elected leader, sending a full tablet report...
I20250814 07:59:38.690850 26642 ts_tablet_manager.cc:1428] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd: Time spent starting tablet: real 0.044s user 0.040s sys 0.004s
I20250814 07:59:38.690894 26644 raft_consensus.cc:695] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [term 1 LEADER]: Becoming Leader. State: Replica: 6a7a2f82967646d2bc25bdf06c7a8ffd, State: Running, Role: LEADER
I20250814 07:59:38.691670 26644 consensus_queue.cc:237] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7a2f82967646d2bc25bdf06c7a8ffd" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 40155 } }
I20250814 07:59:38.703311 26439 catalog_manager.cc:5582] T 76f6b56b1d2842338bda9f592f77a9f5 P 6a7a2f82967646d2bc25bdf06c7a8ffd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a7a2f82967646d2bc25bdf06c7a8ffd (127.20.120.1). New cstate: current_term: 1 leader_uuid: "6a7a2f82967646d2bc25bdf06c7a8ffd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a7a2f82967646d2bc25bdf06c7a8ffd" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 40155 } health_report { overall_health: HEALTHY } } }
I20250814 07:59:38.741605 20960 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250814 07:59:38.744771 20960 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 6a7a2f82967646d2bc25bdf06c7a8ffd to finish bootstrapping
W20250814 07:59:39.207239 26470 debug-util.cc:398] Leaking SignalData structure 0x7b0800085b40 after lost signal to thread 26407
I20250814 07:59:41.415067 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 26498
I20250814 07:59:41.451102 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 26406
2025-08-14T07:59:41Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7888 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7888 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250814 07:59:41.510493 20960 test_util.cc:276] Using random seed: -1478836412
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (11 ms)
[----------] 1 test from ListTableCliParamTest (11 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-08-14T07:59:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-08-14T07:59:41Z Disabled control of system clock
I20250814 07:59:41.557732 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:46215
--webserver_interface=127.20.120.62
--webserver_port=0
--builtin_ntp_servers=127.20.120.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:46215 with env {}
W20250814 07:59:41.846098 26671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:41.846649 26671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:41.847071 26671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:41.878136 26671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:59:41.878413 26671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:41.878614 26671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:59:41.878806 26671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:59:41.912495 26671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:46215
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:46215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:41.913702 26671 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:41.915169 26671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:41.926196 26677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:41.926826 26678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:43.099481 26680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:43.101768 26679 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1171 milliseconds
I20250814 07:59:43.101895 26671 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:59:43.103111 26671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:43.106215 26671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:43.107637 26671 hybrid_clock.cc:648] HybridClock initialized: now 1755158383107593 us; error 50 us; skew 500 ppm
I20250814 07:59:43.108445 26671 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:43.114796 26671 webserver.cc:480] Webserver started at http://127.20.120.62:38767/ using document root <none> and password file <none>
I20250814 07:59:43.115655 26671 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:43.115885 26671 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:43.116331 26671 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:43.120684 26671 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/instance:
uuid: "103a0c4348a042d7a2982d814b6c11e3"
format_stamp: "Formatted at 2025-08-14 07:59:43 on dist-test-slave-87c1"
I20250814 07:59:43.121811 26671 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal/instance:
uuid: "103a0c4348a042d7a2982d814b6c11e3"
format_stamp: "Formatted at 2025-08-14 07:59:43 on dist-test-slave-87c1"
I20250814 07:59:43.128579 26671 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20250814 07:59:43.133746 26687 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:43.134692 26671 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250814 07:59:43.134994 26671 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
uuid: "103a0c4348a042d7a2982d814b6c11e3"
format_stamp: "Formatted at 2025-08-14 07:59:43 on dist-test-slave-87c1"
I20250814 07:59:43.135308 26671 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:43.181221 26671 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:43.182623 26671 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:43.183024 26671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:43.250066 26671 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:46215
I20250814 07:59:43.250142 26738 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:46215 every 8 connection(s)
I20250814 07:59:43.252759 26671 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/info.pb
I20250814 07:59:43.258064 26739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:43.258436 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 26671
I20250814 07:59:43.258992 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal/instance
I20250814 07:59:43.282585 26739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3: Bootstrap starting.
I20250814 07:59:43.288522 26739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:43.290097 26739 log.cc:826] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:43.294160 26739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3: No bootstrap required, opened a new log
I20250814 07:59:43.310160 26739 raft_consensus.cc:357] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103a0c4348a042d7a2982d814b6c11e3" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } }
I20250814 07:59:43.310745 26739 raft_consensus.cc:383] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:43.310969 26739 raft_consensus.cc:738] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 103a0c4348a042d7a2982d814b6c11e3, State: Initialized, Role: FOLLOWER
I20250814 07:59:43.311645 26739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103a0c4348a042d7a2982d814b6c11e3" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } }
I20250814 07:59:43.312139 26739 raft_consensus.cc:397] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:59:43.312381 26739 raft_consensus.cc:491] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:59:43.312669 26739 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:43.316346 26739 raft_consensus.cc:513] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103a0c4348a042d7a2982d814b6c11e3" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } }
I20250814 07:59:43.316974 26739 leader_election.cc:304] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 103a0c4348a042d7a2982d814b6c11e3; no voters:
I20250814 07:59:43.318635 26739 leader_election.cc:290] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:59:43.319445 26744 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:59:43.321403 26744 raft_consensus.cc:695] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [term 1 LEADER]: Becoming Leader. State: Replica: 103a0c4348a042d7a2982d814b6c11e3, State: Running, Role: LEADER
I20250814 07:59:43.322124 26744 consensus_queue.cc:237] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103a0c4348a042d7a2982d814b6c11e3" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } }
I20250814 07:59:43.323020 26739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 07:59:43.335016 26746 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 103a0c4348a042d7a2982d814b6c11e3. Latest consensus state: current_term: 1 leader_uuid: "103a0c4348a042d7a2982d814b6c11e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103a0c4348a042d7a2982d814b6c11e3" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } } }
I20250814 07:59:43.335258 26745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "103a0c4348a042d7a2982d814b6c11e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "103a0c4348a042d7a2982d814b6c11e3" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } } }
I20250814 07:59:43.335777 26746 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [sys.catalog]: This master's current role is: LEADER
I20250814 07:59:43.335894 26745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3 [sys.catalog]: This master's current role is: LEADER
I20250814 07:59:43.338421 26754 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 07:59:43.349073 26754 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 07:59:43.366099 26754 catalog_manager.cc:1349] Generated new cluster ID: 03ad9a20b8f04e97b90f79d1998207fe
I20250814 07:59:43.366314 26754 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 07:59:43.378356 26754 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 07:59:43.380067 26754 catalog_manager.cc:1506] Loading token signing keys...
I20250814 07:59:43.396412 26754 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 103a0c4348a042d7a2982d814b6c11e3: Generated new TSK 0
I20250814 07:59:43.397320 26754 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250814 07:59:43.412547 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:0
--local_ip_for_outbound_sockets=127.20.120.1
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:46215
--builtin_ntp_servers=127.20.120.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250814 07:59:43.700130 26763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:43.700579 26763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:43.701061 26763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:43.730077 26763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:43.730847 26763 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 07:59:43.763085 26763 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=0
--tserver_master_addrs=127.20.120.62:46215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:43.764313 26763 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:43.765821 26763 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:43.777339 26769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:43.778316 26770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:45.175815 26772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:45.180802 26771 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1402 milliseconds
I20250814 07:59:45.180876 26763 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:59:45.182159 26763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:45.184720 26763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:45.186164 26763 hybrid_clock.cc:648] HybridClock initialized: now 1755158385186113 us; error 56 us; skew 500 ppm
I20250814 07:59:45.186897 26763 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:45.193576 26763 webserver.cc:480] Webserver started at http://127.20.120.1:34781/ using document root <none> and password file <none>
I20250814 07:59:45.194438 26763 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:45.194648 26763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:45.195071 26763 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:45.199208 26763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/instance:
uuid: "869fdc28050445f09449e24259230360"
format_stamp: "Formatted at 2025-08-14 07:59:45 on dist-test-slave-87c1"
I20250814 07:59:45.200259 26763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal/instance:
uuid: "869fdc28050445f09449e24259230360"
format_stamp: "Formatted at 2025-08-14 07:59:45 on dist-test-slave-87c1"
I20250814 07:59:45.207317 26763 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20250814 07:59:45.212915 26779 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:45.213930 26763 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250814 07:59:45.214214 26763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal
uuid: "869fdc28050445f09449e24259230360"
format_stamp: "Formatted at 2025-08-14 07:59:45 on dist-test-slave-87c1"
I20250814 07:59:45.214502 26763 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:45.266198 26763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:45.267474 26763 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:45.267890 26763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:45.270923 26763 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:59:45.274771 26763 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:59:45.274943 26763 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:45.275182 26763 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:59:45.275317 26763 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:45.436326 26763 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:39431
I20250814 07:59:45.436470 26891 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:39431 every 8 connection(s)
I20250814 07:59:45.438906 26763 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/info.pb
I20250814 07:59:45.444684 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 26763
I20250814 07:59:45.445228 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal/instance
I20250814 07:59:45.452121 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.2:0
--local_ip_for_outbound_sockets=127.20.120.2
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:46215
--builtin_ntp_servers=127.20.120.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250814 07:59:45.467432 26892 heartbeater.cc:344] Connected to a master server at 127.20.120.62:46215
I20250814 07:59:45.467978 26892 heartbeater.cc:461] Registering TS with master...
I20250814 07:59:45.469347 26892 heartbeater.cc:507] Master 127.20.120.62:46215 requested a full tablet report, sending...
I20250814 07:59:45.472345 26704 ts_manager.cc:194] Registered new tserver with Master: 869fdc28050445f09449e24259230360 (127.20.120.1:39431)
I20250814 07:59:45.475061 26704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:42529
W20250814 07:59:45.746671 26896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:45.747184 26896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:45.747761 26896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:45.777140 26896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:45.777928 26896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.2
I20250814 07:59:45.809900 26896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.2
--webserver_port=0
--tserver_master_addrs=127.20.120.62:46215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.2
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:45.811111 26896 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:45.812667 26896 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:45.823130 26902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:59:46.479238 26892 heartbeater.cc:499] Master 127.20.120.62:46215 was elected leader, sending a full tablet report...
W20250814 07:59:45.824779 26903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:47.153788 26905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:47.157315 26904 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1332 milliseconds
W20250814 07:59:47.158202 26896 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.334s user 0.460s sys 0.859s
W20250814 07:59:47.158579 26896 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.334s user 0.460s sys 0.859s
I20250814 07:59:47.158875 26896 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:59:47.160450 26896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:47.163257 26896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:47.164793 26896 hybrid_clock.cc:648] HybridClock initialized: now 1755158387164742 us; error 43 us; skew 500 ppm
I20250814 07:59:47.166011 26896 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:47.174521 26896 webserver.cc:480] Webserver started at http://127.20.120.2:44595/ using document root <none> and password file <none>
I20250814 07:59:47.175916 26896 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:47.176232 26896 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:47.176919 26896 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:47.184072 26896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/instance:
uuid: "45fe6a2207784f89ba2ff3a48c835c28"
format_stamp: "Formatted at 2025-08-14 07:59:47 on dist-test-slave-87c1"
I20250814 07:59:47.185657 26896 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal/instance:
uuid: "45fe6a2207784f89ba2ff3a48c835c28"
format_stamp: "Formatted at 2025-08-14 07:59:47 on dist-test-slave-87c1"
I20250814 07:59:47.195067 26896 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.002s
I20250814 07:59:47.202272 26912 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:47.203348 26896 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250814 07:59:47.203722 26896 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal
uuid: "45fe6a2207784f89ba2ff3a48c835c28"
format_stamp: "Formatted at 2025-08-14 07:59:47 on dist-test-slave-87c1"
I20250814 07:59:47.204113 26896 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:47.294680 26896 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:47.296021 26896 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:47.296396 26896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:47.298821 26896 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:59:47.302557 26896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:59:47.302742 26896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:47.302929 26896 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:59:47.303061 26896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:47.428488 26896 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.2:36955
I20250814 07:59:47.428578 27024 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.2:36955 every 8 connection(s)
I20250814 07:59:47.430919 26896 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/info.pb
I20250814 07:59:47.433282 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 26896
I20250814 07:59:47.433772 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal/instance
I20250814 07:59:47.444206 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.3:0
--local_ip_for_outbound_sockets=127.20.120.3
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:46215
--builtin_ntp_servers=127.20.120.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250814 07:59:47.471225 27025 heartbeater.cc:344] Connected to a master server at 127.20.120.62:46215
I20250814 07:59:47.471678 27025 heartbeater.cc:461] Registering TS with master...
I20250814 07:59:47.472600 27025 heartbeater.cc:507] Master 127.20.120.62:46215 requested a full tablet report, sending...
I20250814 07:59:47.474783 26704 ts_manager.cc:194] Registered new tserver with Master: 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2:36955)
I20250814 07:59:47.476725 26704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.2:50483
W20250814 07:59:47.741261 27029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:47.741669 27029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:47.742094 27029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:47.771529 27029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:47.772296 27029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.3
I20250814 07:59:47.803882 27029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.3
--webserver_port=0
--tserver_master_addrs=127.20.120.62:46215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.3
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:47.805028 27029 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:47.806576 27029 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:47.818094 27035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:59:48.479806 27025 heartbeater.cc:499] Master 127.20.120.62:46215 was elected leader, sending a full tablet report...
W20250814 07:59:47.818715 27036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:49.174607 27037 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1351 milliseconds
W20250814 07:59:49.176519 27038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:49.178761 27029 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.360s user 0.003s sys 0.006s
W20250814 07:59:49.179118 27029 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.361s user 0.003s sys 0.006s
I20250814 07:59:49.179414 27029 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 07:59:49.180914 27029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 07:59:49.183681 27029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 07:59:49.185168 27029 hybrid_clock.cc:648] HybridClock initialized: now 1755158389185109 us; error 45 us; skew 500 ppm
I20250814 07:59:49.186329 27029 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:49.195047 27029 webserver.cc:480] Webserver started at http://127.20.120.3:36601/ using document root <none> and password file <none>
I20250814 07:59:49.196563 27029 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:49.196877 27029 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:49.197548 27029 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:49.204641 27029 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/instance:
uuid: "f7275b96e2a946de9586a99c02f94622"
format_stamp: "Formatted at 2025-08-14 07:59:49 on dist-test-slave-87c1"
I20250814 07:59:49.206193 27029 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal/instance:
uuid: "f7275b96e2a946de9586a99c02f94622"
format_stamp: "Formatted at 2025-08-14 07:59:49 on dist-test-slave-87c1"
I20250814 07:59:49.215759 27029 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.006s sys 0.004s
I20250814 07:59:49.223433 27045 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:49.224547 27029 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250814 07:59:49.224932 27029 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal
uuid: "f7275b96e2a946de9586a99c02f94622"
format_stamp: "Formatted at 2025-08-14 07:59:49 on dist-test-slave-87c1"
I20250814 07:59:49.225404 27029 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:49.306334 27029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:49.308313 27029 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:49.308892 27029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:49.312340 27029 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 07:59:49.318150 27029 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250814 07:59:49.318332 27029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:49.318528 27029 ts_tablet_manager.cc:610] Registered 0 tablets
I20250814 07:59:49.318656 27029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:49.452572 27029 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.3:44729
I20250814 07:59:49.452682 27157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.3:44729 every 8 connection(s)
I20250814 07:59:49.455179 27029 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/info.pb
I20250814 07:59:49.456249 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 27029
I20250814 07:59:49.456827 20960 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal/instance
I20250814 07:59:49.478811 27158 heartbeater.cc:344] Connected to a master server at 127.20.120.62:46215
I20250814 07:59:49.479200 27158 heartbeater.cc:461] Registering TS with master...
I20250814 07:59:49.480098 27158 heartbeater.cc:507] Master 127.20.120.62:46215 requested a full tablet report, sending...
I20250814 07:59:49.481972 26704 ts_manager.cc:194] Registered new tserver with Master: f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729)
I20250814 07:59:49.483160 26704 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.3:47301
I20250814 07:59:49.491482 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 07:59:49.515183 20960 test_util.cc:276] Using random seed: -1470831721
I20250814 07:59:49.550967 26704 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46970:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250814 07:59:49.553200 26704 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 07:59:49.597322 26827 tablet_service.cc:1468] Processing CreateTablet for tablet cbe9104e14e74b6d8c446ca3077b2952 (DEFAULT_TABLE table=pre_rebuild [id=40e1cccaca15422b9e30ac05abf53267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:59:49.599238 26827 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe9104e14e74b6d8c446ca3077b2952. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:49.602417 26960 tablet_service.cc:1468] Processing CreateTablet for tablet cbe9104e14e74b6d8c446ca3077b2952 (DEFAULT_TABLE table=pre_rebuild [id=40e1cccaca15422b9e30ac05abf53267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:59:49.604157 26960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe9104e14e74b6d8c446ca3077b2952. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:49.604907 27093 tablet_service.cc:1468] Processing CreateTablet for tablet cbe9104e14e74b6d8c446ca3077b2952 (DEFAULT_TABLE table=pre_rebuild [id=40e1cccaca15422b9e30ac05abf53267]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 07:59:49.606664 27093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbe9104e14e74b6d8c446ca3077b2952. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:49.622366 27182 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Bootstrap starting.
I20250814 07:59:49.627604 27182 tablet_bootstrap.cc:654] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:49.629767 27182 log.cc:826] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:49.632510 27183 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Bootstrap starting.
I20250814 07:59:49.635295 27185 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Bootstrap starting.
I20250814 07:59:49.636847 27182 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: No bootstrap required, opened a new log
I20250814 07:59:49.637243 27182 ts_tablet_manager.cc:1397] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20250814 07:59:49.640244 27183 tablet_bootstrap.cc:654] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:49.642617 27183 log.cc:826] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:49.643644 27185 tablet_bootstrap.cc:654] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:49.645869 27185 log.cc:826] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:49.648301 27183 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: No bootstrap required, opened a new log
I20250814 07:59:49.648671 27183 ts_tablet_manager.cc:1397] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.007s
I20250814 07:59:49.651105 27185 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: No bootstrap required, opened a new log
I20250814 07:59:49.651578 27185 ts_tablet_manager.cc:1397] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20250814 07:59:49.657191 27182 raft_consensus.cc:357] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 07:59:49.657826 27182 raft_consensus.cc:383] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:49.658056 27182 raft_consensus.cc:738] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869fdc28050445f09449e24259230360, State: Initialized, Role: FOLLOWER
I20250814 07:59:49.658756 27182 consensus_queue.cc:260] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 07:59:49.662437 27182 ts_tablet_manager.cc:1428] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Time spent starting tablet: real 0.025s user 0.019s sys 0.005s
I20250814 07:59:49.673300 27185 raft_consensus.cc:357] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 07:59:49.673744 27183 raft_consensus.cc:357] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 07:59:49.674149 27185 raft_consensus.cc:383] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:49.674389 27183 raft_consensus.cc:383] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:49.674453 27185 raft_consensus.cc:738] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7275b96e2a946de9586a99c02f94622, State: Initialized, Role: FOLLOWER
I20250814 07:59:49.674635 27183 raft_consensus.cc:738] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fe6a2207784f89ba2ff3a48c835c28, State: Initialized, Role: FOLLOWER
I20250814 07:59:49.675405 27185 consensus_queue.cc:260] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 07:59:49.675421 27183 consensus_queue.cc:260] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 07:59:49.678336 27158 heartbeater.cc:499] Master 127.20.120.62:46215 was elected leader, sending a full tablet report...
I20250814 07:59:49.678963 27183 ts_tablet_manager.cc:1428] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Time spent starting tablet: real 0.030s user 0.024s sys 0.007s
I20250814 07:59:49.679625 27185 ts_tablet_manager.cc:1428] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Time spent starting tablet: real 0.028s user 0.025s sys 0.003s
W20250814 07:59:49.702874 26893 tablet.cc:2378] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 07:59:49.709774 27159 tablet.cc:2378] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 07:59:49.710800 27026 tablet.cc:2378] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 07:59:49.950631 27188 raft_consensus.cc:491] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:59:49.951138 27188 raft_consensus.cc:513] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 07:59:49.953552 27188 leader_election.cc:290] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2:36955), f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729)
I20250814 07:59:49.957345 27189 raft_consensus.cc:491] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 07:59:49.957928 27189 raft_consensus.cc:513] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 07:59:49.964709 27189 leader_election.cc:290] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 869fdc28050445f09449e24259230360 (127.20.120.1:39431), 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2:36955)
I20250814 07:59:49.970865 27113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "869fdc28050445f09449e24259230360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7275b96e2a946de9586a99c02f94622" is_pre_election: true
I20250814 07:59:49.971339 26980 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "869fdc28050445f09449e24259230360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45fe6a2207784f89ba2ff3a48c835c28" is_pre_election: true
I20250814 07:59:49.971866 27113 raft_consensus.cc:2466] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869fdc28050445f09449e24259230360 in term 0.
I20250814 07:59:49.972463 26980 raft_consensus.cc:2466] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 869fdc28050445f09449e24259230360 in term 0.
I20250814 07:59:49.973515 26782 leader_election.cc:304] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869fdc28050445f09449e24259230360, f7275b96e2a946de9586a99c02f94622; no voters:
I20250814 07:59:49.974334 27188 raft_consensus.cc:2802] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 07:59:49.974670 27188 raft_consensus.cc:491] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 07:59:49.974987 27188 raft_consensus.cc:3058] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:49.977887 26847 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "f7275b96e2a946de9586a99c02f94622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869fdc28050445f09449e24259230360" is_pre_election: true
I20250814 07:59:49.980216 27188 raft_consensus.cc:513] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 07:59:49.981195 26847 raft_consensus.cc:2391] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f7275b96e2a946de9586a99c02f94622 in current term 1: Already voted for candidate 869fdc28050445f09449e24259230360 in this term.
I20250814 07:59:49.982944 27188 leader_election.cc:290] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [CANDIDATE]: Term 1 election: Requested vote from peers 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2:36955), f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729)
I20250814 07:59:49.983095 26980 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "869fdc28050445f09449e24259230360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45fe6a2207784f89ba2ff3a48c835c28"
I20250814 07:59:49.983619 26980 raft_consensus.cc:3058] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:49.983968 27113 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "869fdc28050445f09449e24259230360" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7275b96e2a946de9586a99c02f94622"
I20250814 07:59:49.983970 26979 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "f7275b96e2a946de9586a99c02f94622" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45fe6a2207784f89ba2ff3a48c835c28" is_pre_election: true
I20250814 07:59:49.984498 27113 raft_consensus.cc:3058] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:49.988912 27049 leader_election.cc:304] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f7275b96e2a946de9586a99c02f94622; no voters: 45fe6a2207784f89ba2ff3a48c835c28, 869fdc28050445f09449e24259230360
I20250814 07:59:49.989571 27113 raft_consensus.cc:2466] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869fdc28050445f09449e24259230360 in term 1.
I20250814 07:59:49.990214 26980 raft_consensus.cc:2466] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 869fdc28050445f09449e24259230360 in term 1.
I20250814 07:59:49.990386 27189 raft_consensus.cc:2747] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250814 07:59:49.990861 26782 leader_election.cc:304] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 869fdc28050445f09449e24259230360, f7275b96e2a946de9586a99c02f94622; no voters:
I20250814 07:59:49.991451 27188 raft_consensus.cc:2802] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:59:49.992856 27188 raft_consensus.cc:695] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 LEADER]: Becoming Leader. State: Replica: 869fdc28050445f09449e24259230360, State: Running, Role: LEADER
I20250814 07:59:49.993631 27188 consensus_queue.cc:237] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 07:59:50.001466 26703 catalog_manager.cc:5582] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 reported cstate change: term changed from 0 to 1, leader changed from <none> to 869fdc28050445f09449e24259230360 (127.20.120.1). New cstate: current_term: 1 leader_uuid: "869fdc28050445f09449e24259230360" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } health_report { overall_health: UNKNOWN } } }
I20250814 07:59:50.187836 27113 raft_consensus.cc:1273] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 1 FOLLOWER]: Refusing update from remote peer 869fdc28050445f09449e24259230360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250814 07:59:50.188161 26980 raft_consensus.cc:1273] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Refusing update from remote peer 869fdc28050445f09449e24259230360: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250814 07:59:50.189324 27195 consensus_queue.cc:1035] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 07:59:50.189920 27188 consensus_queue.cc:1035] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 07:59:50.215268 27203 mvcc.cc:204] Tried to move back new op lower bound from 7189128766195109888 to 7189128765420388352. Current Snapshot: MvccSnapshot[applied={T|T < 7189128766195109888}]
I20250814 07:59:50.222817 27206 mvcc.cc:204] Tried to move back new op lower bound from 7189128766195109888 to 7189128765420388352. Current Snapshot: MvccSnapshot[applied={T|T < 7189128766195109888}]
I20250814 07:59:50.225090 27205 mvcc.cc:204] Tried to move back new op lower bound from 7189128766195109888 to 7189128765420388352. Current Snapshot: MvccSnapshot[applied={T|T < 7189128766195109888}]
W20250814 07:59:52.477372 27199 meta_cache.cc:1261] Time spent looking up entry by key: real 0.053s user 0.003s sys 0.000s
I20250814 07:59:55.182369 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 26671
W20250814 07:59:55.290967 27025 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:46215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:46215: connect: Connection refused (error 111)
W20250814 07:59:55.300027 26892 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:46215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:46215: connect: Connection refused (error 111)
W20250814 07:59:55.302106 27158 heartbeater.cc:646] Failed to heartbeat to 127.20.120.62:46215 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.20.120.62:46215: connect: Connection refused (error 111)
W20250814 07:59:55.597290 27242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:55.597852 27242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:55.628327 27242 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250814 07:59:57.071800 27250 debug-util.cc:398] Leaking SignalData structure 0x7b08000373e0 after lost signal to thread 27242
W20250814 07:59:57.072515 27250 kernel_stack_watchdog.cc:198] Thread 27242 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 07:59:57.101745 27242 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.433s user 0.002s sys 0.000s
W20250814 07:59:57.206103 27242 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.537s user 0.008s sys 0.007s
I20250814 07:59:57.287204 27242 minidump.cc:252] Setting minidump size limit to 20M
I20250814 07:59:57.289399 27242 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:57.290732 27242 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:57.303735 27275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:57.305907 27276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:57.308527 27279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 07:59:57.308909 27242 server_base.cc:1047] running on GCE node
I20250814 07:59:57.310312 27242 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250814 07:59:57.310825 27242 hybrid_clock.cc:648] HybridClock initialized: now 1755158397310804 us; error 271516 us; skew 500 ppm
I20250814 07:59:57.311501 27242 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 07:59:57.316079 27242 webserver.cc:480] Webserver started at http://0.0.0.0:35077/ using document root <none> and password file <none>
I20250814 07:59:57.317173 27242 fs_manager.cc:362] Metadata directory not provided
I20250814 07:59:57.317412 27242 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 07:59:57.317816 27242 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250814 07:59:57.323397 27242 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/instance:
uuid: "9e74f1f57fd1401c8896fdea89b95c85"
format_stamp: "Formatted at 2025-08-14 07:59:57 on dist-test-slave-87c1"
I20250814 07:59:57.324615 27242 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal/instance:
uuid: "9e74f1f57fd1401c8896fdea89b95c85"
format_stamp: "Formatted at 2025-08-14 07:59:57 on dist-test-slave-87c1"
I20250814 07:59:57.332615 27242 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250814 07:59:57.338389 27286 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 07:59:57.339314 27242 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20250814 07:59:57.339672 27242 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
uuid: "9e74f1f57fd1401c8896fdea89b95c85"
format_stamp: "Formatted at 2025-08-14 07:59:57 on dist-test-slave-87c1"
I20250814 07:59:57.340075 27242 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 07:59:57.590478 27242 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 07:59:57.591689 27242 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 07:59:57.592085 27242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 07:59:57.596349 27242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 07:59:57.609241 27242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: Bootstrap starting.
I20250814 07:59:57.613660 27242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: Neither blocks nor log segments found. Creating new log.
I20250814 07:59:57.615136 27242 log.cc:826] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: Log is configured to *not* fsync() on all Append() calls
I20250814 07:59:57.618768 27242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: No bootstrap required, opened a new log
I20250814 07:59:57.633667 27242 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER }
I20250814 07:59:57.634088 27242 raft_consensus.cc:383] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 07:59:57.634268 27242 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e74f1f57fd1401c8896fdea89b95c85, State: Initialized, Role: FOLLOWER
I20250814 07:59:57.634809 27242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER }
I20250814 07:59:57.635215 27242 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 07:59:57.635432 27242 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 07:59:57.635676 27242 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 0 FOLLOWER]: Advancing to term 1
I20250814 07:59:57.639288 27242 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER }
I20250814 07:59:57.639847 27242 leader_election.cc:304] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e74f1f57fd1401c8896fdea89b95c85; no voters:
I20250814 07:59:57.641440 27242 leader_election.cc:290] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250814 07:59:57.641686 27293 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 07:59:57.644279 27293 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 1 LEADER]: Becoming Leader. State: Replica: 9e74f1f57fd1401c8896fdea89b95c85, State: Running, Role: LEADER
I20250814 07:59:57.645088 27293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER }
I20250814 07:59:57.652467 27294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e74f1f57fd1401c8896fdea89b95c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER } }
I20250814 07:59:57.652573 27295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e74f1f57fd1401c8896fdea89b95c85. Latest consensus state: current_term: 1 leader_uuid: "9e74f1f57fd1401c8896fdea89b95c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER } }
I20250814 07:59:57.652908 27294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [sys.catalog]: This master's current role is: LEADER
I20250814 07:59:57.653162 27295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [sys.catalog]: This master's current role is: LEADER
I20250814 07:59:57.663547 27242 tablet_replica.cc:331] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: stopping tablet replica
I20250814 07:59:57.664068 27242 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 1 LEADER]: Raft consensus shutting down.
I20250814 07:59:57.664466 27242 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250814 07:59:57.666265 27242 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250814 07:59:57.666667 27242 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250814 07:59:57.715641 27242 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250814 07:59:58.748958 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 26763
I20250814 07:59:58.794584 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 26896
I20250814 07:59:58.827136 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 27029
I20250814 07:59:58.860131 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:46215
--webserver_interface=127.20.120.62
--webserver_port=38767
--builtin_ntp_servers=127.20.120.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.20.120.62:46215 with env {}
W20250814 07:59:59.148485 27303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 07:59:59.149031 27303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 07:59:59.149459 27303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 07:59:59.177738 27303 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250814 07:59:59.178022 27303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 07:59:59.178282 27303 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250814 07:59:59.178529 27303 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250814 07:59:59.213490 27303 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.20.120.62:46215
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.20.120.62:46215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.20.120.62
--webserver_port=38767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 07:59:59.214785 27303 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 07:59:59.216356 27303 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 07:59:59.226405 27309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 07:59:59.227108 27310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 08:00:00.630370 27308 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 27303
W20250814 08:00:00.983070 27303 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.756s user 0.570s sys 1.176s
W20250814 08:00:00.983760 27303 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.757s user 0.570s sys 1.176s
W20250814 08:00:00.986100 27312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 08:00:00.988721 27311 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1761 milliseconds
I20250814 08:00:00.988759 27303 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 08:00:00.990031 27303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 08:00:00.992827 27303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 08:00:00.994246 27303 hybrid_clock.cc:648] HybridClock initialized: now 1755158400994206 us; error 58 us; skew 500 ppm
I20250814 08:00:00.995085 27303 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 08:00:01.001542 27303 webserver.cc:480] Webserver started at http://127.20.120.62:38767/ using document root <none> and password file <none>
I20250814 08:00:01.002490 27303 fs_manager.cc:362] Metadata directory not provided
I20250814 08:00:01.002734 27303 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 08:00:01.010761 27303 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.003s
I20250814 08:00:01.015429 27320 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 08:00:01.016489 27303 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250814 08:00:01.016849 27303 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
uuid: "9e74f1f57fd1401c8896fdea89b95c85"
format_stamp: "Formatted at 2025-08-14 07:59:57 on dist-test-slave-87c1"
I20250814 08:00:01.019294 27303 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 08:00:01.078457 27303 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 08:00:01.079979 27303 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 08:00:01.080461 27303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 08:00:01.152551 27303 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.62:46215
I20250814 08:00:01.152621 27371 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.62:46215 every 8 connection(s)
I20250814 08:00:01.155579 27303 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/info.pb
I20250814 08:00:01.165118 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 27303
I20250814 08:00:01.165591 27372 sys_catalog.cc:263] Verifying existing consensus state
I20250814 08:00:01.166941 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.1:39431
--local_ip_for_outbound_sockets=127.20.120.1
--tserver_master_addrs=127.20.120.62:46215
--webserver_port=34781
--webserver_interface=127.20.120.1
--builtin_ntp_servers=127.20.120.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250814 08:00:01.178756 27372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: Bootstrap starting.
I20250814 08:00:01.194509 27372 log.cc:826] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: Log is configured to *not* fsync() on all Append() calls
I20250814 08:00:01.210842 27372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 08:00:01.211894 27372 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: Bootstrap complete.
I20250814 08:00:01.243227 27372 raft_consensus.cc:357] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } }
I20250814 08:00:01.244104 27372 raft_consensus.cc:738] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e74f1f57fd1401c8896fdea89b95c85, State: Initialized, Role: FOLLOWER
I20250814 08:00:01.245052 27372 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } }
I20250814 08:00:01.245746 27372 raft_consensus.cc:397] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250814 08:00:01.246083 27372 raft_consensus.cc:491] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250814 08:00:01.246490 27372 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 1 FOLLOWER]: Advancing to term 2
I20250814 08:00:01.252753 27372 raft_consensus.cc:513] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } }
I20250814 08:00:01.253731 27372 leader_election.cc:304] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e74f1f57fd1401c8896fdea89b95c85; no voters:
I20250814 08:00:01.255883 27372 leader_election.cc:290] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250814 08:00:01.256358 27376 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 08:00:01.259846 27376 raft_consensus.cc:695] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [term 2 LEADER]: Becoming Leader. State: Replica: 9e74f1f57fd1401c8896fdea89b95c85, State: Running, Role: LEADER
I20250814 08:00:01.260687 27376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } }
I20250814 08:00:01.260921 27372 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [sys.catalog]: configured and running, proceeding with master startup.
I20250814 08:00:01.270859 27377 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9e74f1f57fd1401c8896fdea89b95c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } } }
I20250814 08:00:01.271939 27377 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [sys.catalog]: This master's current role is: LEADER
I20250814 08:00:01.274958 27378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e74f1f57fd1401c8896fdea89b95c85. Latest consensus state: current_term: 2 leader_uuid: "9e74f1f57fd1401c8896fdea89b95c85" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e74f1f57fd1401c8896fdea89b95c85" member_type: VOTER last_known_addr { host: "127.20.120.62" port: 46215 } } }
I20250814 08:00:01.275969 27378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85 [sys.catalog]: This master's current role is: LEADER
I20250814 08:00:01.287889 27383 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250814 08:00:01.308791 27383 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=0739d2d03f58454fad8dbd492415bb16]
I20250814 08:00:01.317062 27383 tablet_loader.cc:96] loaded metadata for tablet cbe9104e14e74b6d8c446ca3077b2952 (table pre_rebuild [id=0739d2d03f58454fad8dbd492415bb16])
I20250814 08:00:01.320390 27383 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250814 08:00:01.350099 27383 catalog_manager.cc:1349] Generated new cluster ID: f3c086e789524be898fc2abcbefd473e
I20250814 08:00:01.350482 27383 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250814 08:00:01.370507 27383 catalog_manager.cc:1372] Generated new certificate authority record
I20250814 08:00:01.372573 27383 catalog_manager.cc:1506] Loading token signing keys...
I20250814 08:00:01.398985 27383 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: Generated new TSK 0
I20250814 08:00:01.400190 27383 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250814 08:00:01.564671 27374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 08:00:01.565268 27374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 08:00:01.565793 27374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 08:00:01.598232 27374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 08:00:01.599117 27374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.1
I20250814 08:00:01.635612 27374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.1:39431
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.20.120.1
--webserver_port=34781
--tserver_master_addrs=127.20.120.62:46215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.1
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 08:00:01.637053 27374 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 08:00:01.638813 27374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 08:00:01.652191 27400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 08:00:03.055609 27399 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 27374
W20250814 08:00:01.654129 27401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 08:00:03.369547 27374 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.717s user 0.585s sys 1.064s
W20250814 08:00:03.372655 27374 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.720s user 0.586s sys 1.065s
I20250814 08:00:03.373929 27374 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250814 08:00:03.374003 27403 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 08:00:03.372949 27402 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1718 milliseconds
I20250814 08:00:03.376689 27374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 08:00:03.379020 27374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 08:00:03.380486 27374 hybrid_clock.cc:648] HybridClock initialized: now 1755158403380442 us; error 63 us; skew 500 ppm
I20250814 08:00:03.381260 27374 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 08:00:03.387411 27374 webserver.cc:480] Webserver started at http://127.20.120.1:34781/ using document root <none> and password file <none>
I20250814 08:00:03.388298 27374 fs_manager.cc:362] Metadata directory not provided
I20250814 08:00:03.388533 27374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 08:00:03.396325 27374 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.001s
I20250814 08:00:03.400928 27410 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 08:00:03.402302 27374 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20250814 08:00:03.402689 27374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal
uuid: "869fdc28050445f09449e24259230360"
format_stamp: "Formatted at 2025-08-14 07:59:45 on dist-test-slave-87c1"
I20250814 08:00:03.405434 27374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 08:00:03.464936 27374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 08:00:03.466302 27374 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 08:00:03.466745 27374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 08:00:03.469697 27374 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 08:00:03.475726 27417 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 08:00:03.482631 27374 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 08:00:03.482826 27374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250814 08:00:03.483096 27374 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 08:00:03.487246 27374 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 08:00:03.487499 27374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.002s sys 0.000s
I20250814 08:00:03.487839 27417 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Bootstrap starting.
I20250814 08:00:03.655798 27374 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.1:39431
I20250814 08:00:03.655952 27523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.1:39431 every 8 connection(s)
I20250814 08:00:03.659595 27374 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/info.pb
I20250814 08:00:03.662261 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 27374
I20250814 08:00:03.664201 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.2:36955
--local_ip_for_outbound_sockets=127.20.120.2
--tserver_master_addrs=127.20.120.62:46215
--webserver_port=44595
--webserver_interface=127.20.120.2
--builtin_ntp_servers=127.20.120.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250814 08:00:03.715739 27524 heartbeater.cc:344] Connected to a master server at 127.20.120.62:46215
I20250814 08:00:03.716243 27524 heartbeater.cc:461] Registering TS with master...
I20250814 08:00:03.717471 27524 heartbeater.cc:507] Master 127.20.120.62:46215 requested a full tablet report, sending...
I20250814 08:00:03.722133 27337 ts_manager.cc:194] Registered new tserver with Master: 869fdc28050445f09449e24259230360 (127.20.120.1:39431)
I20250814 08:00:03.730057 27337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.1:36177
I20250814 08:00:03.828334 27417 log.cc:826] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Log is configured to *not* fsync() on all Append() calls
W20250814 08:00:04.114290 27528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 08:00:04.114777 27528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 08:00:04.115276 27528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 08:00:04.144973 27528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 08:00:04.145788 27528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.2
I20250814 08:00:04.180228 27528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.2:36955
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.20.120.2
--webserver_port=44595
--tserver_master_addrs=127.20.120.62:46215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.2
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 08:00:04.181469 27528 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 08:00:04.182945 27528 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 08:00:04.194895 27535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 08:00:04.734336 27524 heartbeater.cc:499] Master 127.20.120.62:46215 was elected leader, sending a full tablet report...
W20250814 08:00:05.597827 27534 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 27528
W20250814 08:00:05.645618 27534 kernel_stack_watchdog.cc:198] Thread 27528 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 399ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250814 08:00:05.646601 27528 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.450s user 0.410s sys 0.941s
W20250814 08:00:05.647090 27528 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.451s user 0.410s sys 0.941s
W20250814 08:00:04.196130 27536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 08:00:05.648829 27538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 08:00:05.651710 27537 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1451 milliseconds
I20250814 08:00:05.651721 27528 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250814 08:00:05.653030 27528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 08:00:05.661589 27528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 08:00:05.663095 27528 hybrid_clock.cc:648] HybridClock initialized: now 1755158405663062 us; error 38 us; skew 500 ppm
I20250814 08:00:05.663834 27528 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 08:00:05.670475 27528 webserver.cc:480] Webserver started at http://127.20.120.2:44595/ using document root <none> and password file <none>
I20250814 08:00:05.671365 27528 fs_manager.cc:362] Metadata directory not provided
I20250814 08:00:05.671597 27528 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 08:00:05.678983 27528 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20250814 08:00:05.683893 27545 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 08:00:05.684934 27528 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250814 08:00:05.685279 27528 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal
uuid: "45fe6a2207784f89ba2ff3a48c835c28"
format_stamp: "Formatted at 2025-08-14 07:59:47 on dist-test-slave-87c1"
I20250814 08:00:05.687011 27528 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 08:00:05.738839 27528 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 08:00:05.740254 27528 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 08:00:05.740664 27528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 08:00:05.743542 27528 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 08:00:05.750105 27552 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 08:00:05.757261 27528 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 08:00:05.757448 27528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.002s sys 0.000s
I20250814 08:00:05.757690 27528 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 08:00:05.761890 27528 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 08:00:05.762128 27528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20250814 08:00:05.762571 27552 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Bootstrap starting.
I20250814 08:00:05.931993 27528 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.2:36955
I20250814 08:00:05.932138 27658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.2:36955 every 8 connection(s)
I20250814 08:00:05.935703 27528 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/info.pb
I20250814 08:00:05.938882 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 27528
I20250814 08:00:05.940753 20960 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
/tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.20.120.3:44729
--local_ip_for_outbound_sockets=127.20.120.3
--tserver_master_addrs=127.20.120.62:46215
--webserver_port=36601
--webserver_interface=127.20.120.3
--builtin_ntp_servers=127.20.120.20:41449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250814 08:00:05.986997 27659 heartbeater.cc:344] Connected to a master server at 127.20.120.62:46215
I20250814 08:00:05.987617 27659 heartbeater.cc:461] Registering TS with master...
I20250814 08:00:05.988818 27659 heartbeater.cc:507] Master 127.20.120.62:46215 requested a full tablet report, sending...
I20250814 08:00:05.992563 27337 ts_manager.cc:194] Registered new tserver with Master: 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2:36955)
I20250814 08:00:05.995524 27337 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.2:38925
I20250814 08:00:06.069996 27552 log.cc:826] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Log is configured to *not* fsync() on all Append() calls
W20250814 08:00:06.386826 27663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250814 08:00:06.387456 27663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250814 08:00:06.388242 27663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250814 08:00:06.441911 27663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250814 08:00:06.443241 27663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.20.120.3
I20250814 08:00:06.502504 27663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.20.120.20:41449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.20.120.3:44729
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.20.120.3
--webserver_port=36601
--tserver_master_addrs=127.20.120.62:46215
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.20.120.3
--log_dir=/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 6307ab29b78ebc064971e01ca1da31590298dcda
build type FASTDEBUG
built by None at 14 Aug 2025 07:43:22 UTC on 5fd53c4cbb9d
build id 7561
TSAN enabled
I20250814 08:00:06.504048 27663 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250814 08:00:06.506119 27663 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250814 08:00:06.522475 27671 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 08:00:06.999037 27659 heartbeater.cc:499] Master 127.20.120.62:46215 was elected leader, sending a full tablet report...
I20250814 08:00:07.300551 27417 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 08:00:07.301954 27417 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Bootstrap complete.
I20250814 08:00:07.304322 27417 ts_tablet_manager.cc:1397] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Time spent bootstrapping tablet: real 3.817s user 3.745s sys 0.056s
I20250814 08:00:07.326458 27417 raft_consensus.cc:357] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 08:00:07.330420 27417 raft_consensus.cc:738] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869fdc28050445f09449e24259230360, State: Initialized, Role: FOLLOWER
I20250814 08:00:07.331539 27417 consensus_queue.cc:260] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 08:00:07.341305 27417 ts_tablet_manager.cc:1428] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Time spent starting tablet: real 0.037s user 0.028s sys 0.008s
W20250814 08:00:07.922185 27669 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 27663
W20250814 08:00:06.523491 27670 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250814 08:00:08.144874 27663 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.625s user 0.584s sys 0.966s
W20250814 08:00:08.145540 27663 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.626s user 0.585s sys 0.966s
W20250814 08:00:08.153172 27676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250814 08:00:08.153303 27663 server_base.cc:1047] running on GCE node
I20250814 08:00:08.154587 27663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250814 08:00:08.156883 27663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250814 08:00:08.158409 27663 hybrid_clock.cc:648] HybridClock initialized: now 1755158408158384 us; error 50 us; skew 500 ppm
I20250814 08:00:08.159222 27663 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250814 08:00:08.165774 27663 webserver.cc:480] Webserver started at http://127.20.120.3:36601/ using document root <none> and password file <none>
I20250814 08:00:08.166759 27663 fs_manager.cc:362] Metadata directory not provided
I20250814 08:00:08.167006 27663 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250814 08:00:08.174764 27663 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20250814 08:00:08.179528 27681 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250814 08:00:08.180528 27663 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250814 08:00:08.180801 27663 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data,/tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal
uuid: "f7275b96e2a946de9586a99c02f94622"
format_stamp: "Formatted at 2025-08-14 07:59:49 on dist-test-slave-87c1"
I20250814 08:00:08.182574 27663 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250814 08:00:08.235054 27663 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250814 08:00:08.236340 27663 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250814 08:00:08.236752 27663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250814 08:00:08.239639 27663 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250814 08:00:08.245767 27688 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250814 08:00:08.255780 27663 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250814 08:00:08.255996 27663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.000s sys 0.002s
I20250814 08:00:08.256239 27663 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250814 08:00:08.260568 27663 ts_tablet_manager.cc:610] Registered 1 tablets
I20250814 08:00:08.260821 27663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.001s
I20250814 08:00:08.261149 27688 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Bootstrap starting.
I20250814 08:00:08.466009 27663 rpc_server.cc:307] RPC server started. Bound to: 127.20.120.3:44729
I20250814 08:00:08.466251 27794 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.20.120.3:44729 every 8 connection(s)
I20250814 08:00:08.468953 27663 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/info.pb
I20250814 08:00:08.472702 20960 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu as pid 27663
I20250814 08:00:08.500083 27796 raft_consensus.cc:491] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 08:00:08.500674 27796 raft_consensus.cc:513] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 08:00:08.517668 27796 leader_election.cc:290] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2:36955), f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729)
I20250814 08:00:08.525964 27795 heartbeater.cc:344] Connected to a master server at 127.20.120.62:46215
I20250814 08:00:08.526489 27795 heartbeater.cc:461] Registering TS with master...
I20250814 08:00:08.527822 27795 heartbeater.cc:507] Master 127.20.120.62:46215 requested a full tablet report, sending...
I20250814 08:00:08.533921 27336 ts_manager.cc:194] Registered new tserver with Master: f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729)
I20250814 08:00:08.537427 27336 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.20.120.3:40241
I20250814 08:00:08.554106 20960 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250814 08:00:08.547065 27614 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "869fdc28050445f09449e24259230360" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "45fe6a2207784f89ba2ff3a48c835c28" is_pre_election: true
W20250814 08:00:08.578622 27411 leader_election.cc:343] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2:36955): Illegal state: must be running to vote when last-logged opid is not known
I20250814 08:00:08.580093 27552 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 08:00:08.581255 27552 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Bootstrap complete.
I20250814 08:00:08.583040 27552 ts_tablet_manager.cc:1397] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Time spent bootstrapping tablet: real 2.821s user 2.703s sys 0.069s
I20250814 08:00:08.588330 27688 log.cc:826] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Log is configured to *not* fsync() on all Append() calls
I20250814 08:00:08.580149 27750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "869fdc28050445f09449e24259230360" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f7275b96e2a946de9586a99c02f94622" is_pre_election: true
I20250814 08:00:08.590834 27552 raft_consensus.cc:357] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
W20250814 08:00:08.593990 27413 leader_election.cc:343] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729): Illegal state: must be running to vote when last-logged opid is not known
I20250814 08:00:08.594388 27552 raft_consensus.cc:738] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fe6a2207784f89ba2ff3a48c835c28, State: Initialized, Role: FOLLOWER
I20250814 08:00:08.594403 27413 leader_election.cc:304] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 869fdc28050445f09449e24259230360; no voters: 45fe6a2207784f89ba2ff3a48c835c28, f7275b96e2a946de9586a99c02f94622
I20250814 08:00:08.595525 27796 raft_consensus.cc:2747] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250814 08:00:08.595288 27552 consensus_queue.cc:260] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 08:00:08.600069 27552 ts_tablet_manager.cc:1428] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20250814 08:00:09.541663 27795 heartbeater.cc:499] Master 127.20.120.62:46215 was elected leader, sending a full tablet report...
I20250814 08:00:09.737097 27814 raft_consensus.cc:491] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 08:00:09.737500 27814 raft_consensus.cc:513] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 08:00:09.739894 27814 leader_election.cc:290] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 869fdc28050445f09449e24259230360 (127.20.120.1:39431), f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729)
I20250814 08:00:09.760521 27479 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "45fe6a2207784f89ba2ff3a48c835c28" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "869fdc28050445f09449e24259230360" is_pre_election: true
I20250814 08:00:09.761214 27479 raft_consensus.cc:2466] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45fe6a2207784f89ba2ff3a48c835c28 in term 1.
I20250814 08:00:09.761404 27750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "45fe6a2207784f89ba2ff3a48c835c28" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f7275b96e2a946de9586a99c02f94622" is_pre_election: true
I20250814 08:00:09.762586 27549 leader_election.cc:304] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45fe6a2207784f89ba2ff3a48c835c28, 869fdc28050445f09449e24259230360; no voters:
W20250814 08:00:09.763381 27548 leader_election.cc:343] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729): Illegal state: must be running to vote when last-logged opid is not known
I20250814 08:00:09.763463 27814 raft_consensus.cc:2802] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250814 08:00:09.763824 27814 raft_consensus.cc:491] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 08:00:09.764149 27814 raft_consensus.cc:3058] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Advancing to term 2
I20250814 08:00:09.772758 27814 raft_consensus.cc:513] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 08:00:09.774456 27814 leader_election.cc:290] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [CANDIDATE]: Term 2 election: Requested vote from peers 869fdc28050445f09449e24259230360 (127.20.120.1:39431), f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729)
I20250814 08:00:09.775053 27479 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "45fe6a2207784f89ba2ff3a48c835c28" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "869fdc28050445f09449e24259230360"
I20250814 08:00:09.775477 27479 raft_consensus.cc:3058] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Advancing to term 2
I20250814 08:00:09.775384 27750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbe9104e14e74b6d8c446ca3077b2952" candidate_uuid: "45fe6a2207784f89ba2ff3a48c835c28" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f7275b96e2a946de9586a99c02f94622"
W20250814 08:00:09.776337 27548 leader_election.cc:343] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729): Illegal state: must be running to vote when last-logged opid is not known
I20250814 08:00:09.780977 27479 raft_consensus.cc:2466] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45fe6a2207784f89ba2ff3a48c835c28 in term 2.
I20250814 08:00:09.781749 27549 leader_election.cc:304] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 45fe6a2207784f89ba2ff3a48c835c28, 869fdc28050445f09449e24259230360; no voters: f7275b96e2a946de9586a99c02f94622
I20250814 08:00:09.782267 27814 raft_consensus.cc:2802] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 2 FOLLOWER]: Leader election won for term 2
I20250814 08:00:09.783610 27814 raft_consensus.cc:695] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 2 LEADER]: Becoming Leader. State: Replica: 45fe6a2207784f89ba2ff3a48c835c28, State: Running, Role: LEADER
I20250814 08:00:09.784476 27814 consensus_queue.cc:237] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 08:00:09.794588 27336 catalog_manager.cc:5582] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 reported cstate change: term changed from 0 to 2, leader changed from <none> to 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2), VOTER 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2) added, VOTER 869fdc28050445f09449e24259230360 (127.20.120.1) added, VOTER f7275b96e2a946de9586a99c02f94622 (127.20.120.3) added. New cstate: current_term: 2 leader_uuid: "45fe6a2207784f89ba2ff3a48c835c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } health_report { overall_health: UNKNOWN } } }
I20250814 08:00:10.207484 27479 raft_consensus.cc:1273] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 2 FOLLOWER]: Refusing update from remote peer 45fe6a2207784f89ba2ff3a48c835c28: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250814 08:00:10.209190 27814 consensus_queue.cc:1035] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.000s
W20250814 08:00:10.289865 27548 consensus_peers.cc:489] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 -> Peer f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729): Couldn't send request to peer f7275b96e2a946de9586a99c02f94622. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250814 08:00:10.328816 27614 consensus_queue.cc:237] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:10.334666 27478 raft_consensus.cc:1273] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 2 FOLLOWER]: Refusing update from remote peer 45fe6a2207784f89ba2ff3a48c835c28: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250814 08:00:10.336318 27833 consensus_queue.cc:1035] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.001s
I20250814 08:00:10.343221 27833 raft_consensus.cc:2953] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER f7275b96e2a946de9586a99c02f94622 (127.20.120.3) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } }
I20250814 08:00:10.345701 27478 raft_consensus.cc:2953] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER f7275b96e2a946de9586a99c02f94622 (127.20.120.3) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } }
I20250814 08:00:10.363168 27321 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet cbe9104e14e74b6d8c446ca3077b2952 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250814 08:00:10.367893 27336 catalog_manager.cc:5582] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 reported cstate change: config changed from index -1 to 207, VOTER f7275b96e2a946de9586a99c02f94622 (127.20.120.3) evicted. New cstate: current_term: 2 leader_uuid: "45fe6a2207784f89ba2ff3a48c835c28" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } health_report { overall_health: HEALTHY } } }
I20250814 08:00:10.398587 27614 consensus_queue.cc:237] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:10.401043 27833 raft_consensus.cc:2953] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER 869fdc28050445f09449e24259230360 (127.20.120.1) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } }
I20250814 08:00:10.413203 27321 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet cbe9104e14e74b6d8c446ca3077b2952 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250814 08:00:10.417359 27336 catalog_manager.cc:5582] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 reported cstate change: config changed from index 207 to 208, VOTER 869fdc28050445f09449e24259230360 (127.20.120.1) evicted. New cstate: current_term: 2 leader_uuid: "45fe6a2207784f89ba2ff3a48c835c28" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } health_report { overall_health: HEALTHY } } }
I20250814 08:00:10.472863 27459 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe9104e14e74b6d8c446ca3077b2952 with delete_type TABLET_DATA_TOMBSTONED (TS 869fdc28050445f09449e24259230360 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:49820
I20250814 08:00:10.488178 27730 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe9104e14e74b6d8c446ca3077b2952 with delete_type TABLET_DATA_TOMBSTONED (TS f7275b96e2a946de9586a99c02f94622 not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:37734
I20250814 08:00:10.497088 27842 tablet_replica.cc:331] stopping tablet replica
W20250814 08:00:10.500190 27323 catalog_manager.cc:4908] TS f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729): delete failed for tablet cbe9104e14e74b6d8c446ca3077b2952 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe9104e14e74b6d8c446ca3077b2952 already in progress: opening tablet
I20250814 08:00:10.501477 27842 raft_consensus.cc:2241] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250814 08:00:10.502107 27842 raft_consensus.cc:2270] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360 [term 2 FOLLOWER]: Raft consensus is shut down!
W20250814 08:00:10.514894 20960 scanner-internal.cc:458] Time spent opening tablet: real 1.893s user 0.006s sys 0.000s
I20250814 08:00:10.531312 27688 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250814 08:00:10.532287 27688 tablet_bootstrap.cc:492] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Bootstrap complete.
I20250814 08:00:10.534030 27688 ts_tablet_manager.cc:1397] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Time spent bootstrapping tablet: real 2.273s user 2.163s sys 0.056s
I20250814 08:00:10.542127 27688 raft_consensus.cc:357] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 08:00:10.545092 27688 raft_consensus.cc:738] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7275b96e2a946de9586a99c02f94622, State: Initialized, Role: FOLLOWER
I20250814 08:00:10.546016 27688 consensus_queue.cc:260] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } } peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } }
I20250814 08:00:10.556810 27688 ts_tablet_manager.cc:1428] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Time spent starting tablet: real 0.022s user 0.018s sys 0.000s
I20250814 08:00:10.561161 27730 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe9104e14e74b6d8c446ca3077b2952 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:37734
W20250814 08:00:10.562750 27323 catalog_manager.cc:4908] TS f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729): delete failed for tablet cbe9104e14e74b6d8c446ca3077b2952 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbe9104e14e74b6d8c446ca3077b2952 already in progress: opening tablet
I20250814 08:00:10.563370 27842 ts_tablet_manager.cc:1905] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250814 08:00:10.571080 27730 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe9104e14e74b6d8c446ca3077b2952 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:37734
I20250814 08:00:10.571928 27843 tablet_replica.cc:331] stopping tablet replica
I20250814 08:00:10.572613 27843 raft_consensus.cc:2241] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250814 08:00:10.573176 27843 raft_consensus.cc:2270] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250814 08:00:10.578233 27842 ts_tablet_manager.cc:1918] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250814 08:00:10.578576 27842 log.cc:1199] T cbe9104e14e74b6d8c446ca3077b2952 P 869fdc28050445f09449e24259230360: Deleting WAL directory at /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/wal/wals/cbe9104e14e74b6d8c446ca3077b2952
I20250814 08:00:10.580461 27324 catalog_manager.cc:4928] TS 869fdc28050445f09449e24259230360 (127.20.120.1:39431): tablet cbe9104e14e74b6d8c446ca3077b2952 (table pre_rebuild [id=0739d2d03f58454fad8dbd492415bb16]) successfully deleted
I20250814 08:00:10.601428 27843 ts_tablet_manager.cc:1905] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250814 08:00:10.617357 27843 ts_tablet_manager.cc:1918] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250814 08:00:10.617713 27843 log.cc:1199] T cbe9104e14e74b6d8c446ca3077b2952 P f7275b96e2a946de9586a99c02f94622: Deleting WAL directory at /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/wal/wals/cbe9104e14e74b6d8c446ca3077b2952
I20250814 08:00:10.619148 27323 catalog_manager.cc:4928] TS f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729): tablet cbe9104e14e74b6d8c446ca3077b2952 (table pre_rebuild [id=0739d2d03f58454fad8dbd492415bb16]) successfully deleted
I20250814 08:00:11.164390 27594 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 08:00:11.182070 27459 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 08:00:11.187841 27730 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
9e74f1f57fd1401c8896fdea89b95c85 | 127.20.120.62:46215 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.120.20:41449 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
45fe6a2207784f89ba2ff3a48c835c28 | 127.20.120.2:36955 | HEALTHY | <none> | 1 | 0
869fdc28050445f09449e24259230360 | 127.20.120.1:39431 | HEALTHY | <none> | 0 | 0
f7275b96e2a946de9586a99c02f94622 | 127.20.120.3:44729 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.120.1 | experimental | 127.20.120.1:39431
local_ip_for_outbound_sockets | 127.20.120.2 | experimental | 127.20.120.2:36955
local_ip_for_outbound_sockets | 127.20.120.3 | experimental | 127.20.120.3:44729
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/info.pb | hidden | 127.20.120.1:39431
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/info.pb | hidden | 127.20.120.2:36955
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/info.pb | hidden | 127.20.120.3:44729
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.120.20:41449 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250814 08:00:11.454439 20960 log_verifier.cc:126] Checking tablet cbe9104e14e74b6d8c446ca3077b2952
I20250814 08:00:11.734350 20960 log_verifier.cc:177] Verified matching terms for 208 ops in tablet cbe9104e14e74b6d8c446ca3077b2952
I20250814 08:00:11.736748 27336 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50256:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250814 08:00:11.737406 27336 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50256:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250814 08:00:11.750103 27336 catalog_manager.cc:5869] T 00000000000000000000000000000000 P 9e74f1f57fd1401c8896fdea89b95c85: Sending DeleteTablet for 1 replicas of tablet cbe9104e14e74b6d8c446ca3077b2952
I20250814 08:00:11.751780 20960 test_util.cc:276] Using random seed: -1448595122
I20250814 08:00:11.751775 27594 tablet_service.cc:1515] Processing DeleteTablet for tablet cbe9104e14e74b6d8c446ca3077b2952 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-08-14 08:00:11 UTC) from {username='slave'} at 127.0.0.1:46960
I20250814 08:00:11.758023 27877 tablet_replica.cc:331] stopping tablet replica
I20250814 08:00:11.758888 27877 raft_consensus.cc:2241] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 2 LEADER]: Raft consensus shutting down.
I20250814 08:00:11.759454 27877 raft_consensus.cc:2270] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250814 08:00:11.785254 27336 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50288:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250814 08:00:11.788487 27336 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250814 08:00:11.793064 27877 ts_tablet_manager.cc:1905] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250814 08:00:11.805341 27877 ts_tablet_manager.cc:1918] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250814 08:00:11.805800 27877 log.cc:1199] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Deleting WAL directory at /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/wal/wals/cbe9104e14e74b6d8c446ca3077b2952
I20250814 08:00:11.806847 27877 ts_tablet_manager.cc:1939] T cbe9104e14e74b6d8c446ca3077b2952 P 45fe6a2207784f89ba2ff3a48c835c28: Deleting consensus metadata
I20250814 08:00:11.809231 27321 catalog_manager.cc:4928] TS 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2:36955): tablet cbe9104e14e74b6d8c446ca3077b2952 (table pre_rebuild [id=0739d2d03f58454fad8dbd492415bb16]) successfully deleted
I20250814 08:00:11.813995 27730 tablet_service.cc:1468] Processing CreateTablet for tablet 9fc807e872c648ff98e8e7901d06f01d (DEFAULT_TABLE table=post_rebuild [id=aca1dc9a951745ee91f2d470d36fa3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 08:00:11.814714 27594 tablet_service.cc:1468] Processing CreateTablet for tablet 9fc807e872c648ff98e8e7901d06f01d (DEFAULT_TABLE table=post_rebuild [id=aca1dc9a951745ee91f2d470d36fa3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 08:00:11.815675 27594 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc807e872c648ff98e8e7901d06f01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 08:00:11.815693 27730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc807e872c648ff98e8e7901d06f01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 08:00:11.819450 27459 tablet_service.cc:1468] Processing CreateTablet for tablet 9fc807e872c648ff98e8e7901d06f01d (DEFAULT_TABLE table=post_rebuild [id=aca1dc9a951745ee91f2d470d36fa3e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250814 08:00:11.821548 27459 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9fc807e872c648ff98e8e7901d06f01d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250814 08:00:11.839660 27885 tablet_bootstrap.cc:492] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622: Bootstrap starting.
I20250814 08:00:11.842142 27884 tablet_bootstrap.cc:492] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28: Bootstrap starting.
I20250814 08:00:11.846626 27885 tablet_bootstrap.cc:654] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622: Neither blocks nor log segments found. Creating new log.
I20250814 08:00:11.850411 27886 tablet_bootstrap.cc:492] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360: Bootstrap starting.
I20250814 08:00:11.855073 27884 tablet_bootstrap.cc:654] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28: Neither blocks nor log segments found. Creating new log.
I20250814 08:00:11.857797 27885 tablet_bootstrap.cc:492] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622: No bootstrap required, opened a new log
I20250814 08:00:11.857851 27886 tablet_bootstrap.cc:654] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360: Neither blocks nor log segments found. Creating new log.
I20250814 08:00:11.858247 27885 ts_tablet_manager.cc:1397] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622: Time spent bootstrapping tablet: real 0.019s user 0.009s sys 0.007s
I20250814 08:00:11.861229 27885 raft_consensus.cc:357] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } } peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:11.862337 27885 raft_consensus.cc:383] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 08:00:11.862737 27885 raft_consensus.cc:738] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7275b96e2a946de9586a99c02f94622, State: Initialized, Role: FOLLOWER
I20250814 08:00:11.863692 27885 consensus_queue.cc:260] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } } peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:11.866407 27886 tablet_bootstrap.cc:492] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360: No bootstrap required, opened a new log
I20250814 08:00:11.866809 27886 ts_tablet_manager.cc:1397] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.002s
I20250814 08:00:11.867353 27884 tablet_bootstrap.cc:492] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28: No bootstrap required, opened a new log
I20250814 08:00:11.867866 27884 ts_tablet_manager.cc:1397] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28: Time spent bootstrapping tablet: real 0.026s user 0.016s sys 0.000s
I20250814 08:00:11.871129 27884 raft_consensus.cc:357] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } } peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:11.871821 27884 raft_consensus.cc:383] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 08:00:11.872144 27884 raft_consensus.cc:738] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45fe6a2207784f89ba2ff3a48c835c28, State: Initialized, Role: FOLLOWER
I20250814 08:00:11.869603 27886 raft_consensus.cc:357] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } } peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:11.872490 27886 raft_consensus.cc:383] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250814 08:00:11.872807 27886 raft_consensus.cc:738] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 869fdc28050445f09449e24259230360, State: Initialized, Role: FOLLOWER
I20250814 08:00:11.872879 27884 consensus_queue.cc:260] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } } peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:11.873456 27885 ts_tablet_manager.cc:1428] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622: Time spent starting tablet: real 0.015s user 0.005s sys 0.007s
I20250814 08:00:11.873577 27886 consensus_queue.cc:260] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } } peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:11.881992 27884 ts_tablet_manager.cc:1428] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28: Time spent starting tablet: real 0.014s user 0.000s sys 0.012s
I20250814 08:00:11.885251 27886 ts_tablet_manager.cc:1428] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360: Time spent starting tablet: real 0.018s user 0.005s sys 0.009s
W20250814 08:00:11.956821 27525 tablet.cc:2378] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 08:00:11.957386 27660 tablet.cc:2378] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250814 08:00:11.987933 27799 tablet.cc:2378] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250814 08:00:12.002585 27891 raft_consensus.cc:491] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250814 08:00:12.002923 27891 raft_consensus.cc:513] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } } peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:12.004472 27891 leader_election.cc:290] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729), 869fdc28050445f09449e24259230360 (127.20.120.1:39431)
I20250814 08:00:12.005275 27750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fc807e872c648ff98e8e7901d06f01d" candidate_uuid: "45fe6a2207784f89ba2ff3a48c835c28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7275b96e2a946de9586a99c02f94622" is_pre_election: true
I20250814 08:00:12.005831 27750 raft_consensus.cc:2466] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45fe6a2207784f89ba2ff3a48c835c28 in term 0.
I20250814 08:00:12.005816 27478 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fc807e872c648ff98e8e7901d06f01d" candidate_uuid: "45fe6a2207784f89ba2ff3a48c835c28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869fdc28050445f09449e24259230360" is_pre_election: true
I20250814 08:00:12.006484 27478 raft_consensus.cc:2466] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45fe6a2207784f89ba2ff3a48c835c28 in term 0.
I20250814 08:00:12.006671 27548 leader_election.cc:304] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45fe6a2207784f89ba2ff3a48c835c28, f7275b96e2a946de9586a99c02f94622; no voters:
I20250814 08:00:12.007244 27891 raft_consensus.cc:2802] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250814 08:00:12.007526 27891 raft_consensus.cc:491] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250814 08:00:12.007807 27891 raft_consensus.cc:3058] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 0 FOLLOWER]: Advancing to term 1
I20250814 08:00:12.011852 27891 raft_consensus.cc:513] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } } peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:12.013152 27891 leader_election.cc:290] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [CANDIDATE]: Term 1 election: Requested vote from peers f7275b96e2a946de9586a99c02f94622 (127.20.120.3:44729), 869fdc28050445f09449e24259230360 (127.20.120.1:39431)
I20250814 08:00:12.013883 27750 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fc807e872c648ff98e8e7901d06f01d" candidate_uuid: "45fe6a2207784f89ba2ff3a48c835c28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7275b96e2a946de9586a99c02f94622"
I20250814 08:00:12.014271 27750 raft_consensus.cc:3058] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622 [term 0 FOLLOWER]: Advancing to term 1
I20250814 08:00:12.014125 27478 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9fc807e872c648ff98e8e7901d06f01d" candidate_uuid: "45fe6a2207784f89ba2ff3a48c835c28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "869fdc28050445f09449e24259230360"
I20250814 08:00:12.014591 27478 raft_consensus.cc:3058] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360 [term 0 FOLLOWER]: Advancing to term 1
I20250814 08:00:12.018605 27750 raft_consensus.cc:2466] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45fe6a2207784f89ba2ff3a48c835c28 in term 1.
I20250814 08:00:12.019474 27548 leader_election.cc:304] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45fe6a2207784f89ba2ff3a48c835c28, f7275b96e2a946de9586a99c02f94622; no voters:
I20250814 08:00:12.020020 27891 raft_consensus.cc:2802] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 FOLLOWER]: Leader election won for term 1
I20250814 08:00:12.021039 27478 raft_consensus.cc:2466] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45fe6a2207784f89ba2ff3a48c835c28 in term 1.
I20250814 08:00:12.021831 27891 raft_consensus.cc:695] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [term 1 LEADER]: Becoming Leader. State: Replica: 45fe6a2207784f89ba2ff3a48c835c28, State: Running, Role: LEADER
I20250814 08:00:12.022819 27891 consensus_queue.cc:237] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } } peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } }
I20250814 08:00:12.030061 27335 catalog_manager.cc:5582] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45fe6a2207784f89ba2ff3a48c835c28 (127.20.120.2). New cstate: current_term: 1 leader_uuid: "45fe6a2207784f89ba2ff3a48c835c28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "45fe6a2207784f89ba2ff3a48c835c28" member_type: VOTER last_known_addr { host: "127.20.120.2" port: 36955 } health_report { overall_health: HEALTHY } } }
I20250814 08:00:12.254814 27478 raft_consensus.cc:1273] T 9fc807e872c648ff98e8e7901d06f01d P 869fdc28050445f09449e24259230360 [term 1 FOLLOWER]: Refusing update from remote peer 45fe6a2207784f89ba2ff3a48c835c28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250814 08:00:12.255535 27750 raft_consensus.cc:1273] T 9fc807e872c648ff98e8e7901d06f01d P f7275b96e2a946de9586a99c02f94622 [term 1 FOLLOWER]: Refusing update from remote peer 45fe6a2207784f89ba2ff3a48c835c28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250814 08:00:12.256446 27893 consensus_queue.cc:1035] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "869fdc28050445f09449e24259230360" member_type: VOTER last_known_addr { host: "127.20.120.1" port: 39431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250814 08:00:12.257318 27891 consensus_queue.cc:1035] T 9fc807e872c648ff98e8e7901d06f01d P 45fe6a2207784f89ba2ff3a48c835c28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7275b96e2a946de9586a99c02f94622" member_type: VOTER last_known_addr { host: "127.20.120.3" port: 44729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250814 08:00:14.117870 27791 debug-util.cc:398] Leaking SignalData structure 0x7b08000af460 after lost signal to thread 27665
W20250814 08:00:14.119007 27791 debug-util.cc:398] Leaking SignalData structure 0x7b08000cbfe0 after lost signal to thread 27794
W20250814 08:00:16.529729 27655 debug-util.cc:398] Leaking SignalData structure 0x7b08000af5e0 after lost signal to thread 27530
W20250814 08:00:16.530849 27655 debug-util.cc:398] Leaking SignalData structure 0x7b08000afbe0 after lost signal to thread 27658
W20250814 08:00:17.075582 27897 meta_cache.cc:1261] Time spent looking up entry by key: real 0.050s user 0.000s sys 0.000s
I20250814 08:00:18.039081 27594 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250814 08:00:18.059535 27459 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250814 08:00:18.080926 27730 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
9e74f1f57fd1401c8896fdea89b95c85 | 127.20.120.62:46215 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.120.20:41449 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+--------------------+---------+----------+----------------+-----------------
45fe6a2207784f89ba2ff3a48c835c28 | 127.20.120.2:36955 | HEALTHY | <none> | 1 | 0
869fdc28050445f09449e24259230360 | 127.20.120.1:39431 | HEALTHY | <none> | 0 | 0
f7275b96e2a946de9586a99c02f94622 | 127.20.120.3:44729 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.20.120.1 | experimental | 127.20.120.1:39431
local_ip_for_outbound_sockets | 127.20.120.2 | experimental | 127.20.120.2:36955
local_ip_for_outbound_sockets | 127.20.120.3 | experimental | 127.20.120.3:44729
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-0/data/info.pb | hidden | 127.20.120.1:39431
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-1/data/info.pb | hidden | 127.20.120.2:36955
server_dump_info_path | /tmp/dist-test-taskqdzHdL/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1755158230225856-20960-0/minicluster-data/ts-2/data/info.pb | hidden | 127.20.120.3:44729
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.20.120.20:41449 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250814 08:00:18.331941 20960 log_verifier.cc:126] Checking tablet 9fc807e872c648ff98e8e7901d06f01d
I20250814 08:00:19.148337 20960 log_verifier.cc:177] Verified matching terms for 207 ops in tablet 9fc807e872c648ff98e8e7901d06f01d
I20250814 08:00:19.149284 20960 log_verifier.cc:126] Checking tablet cbe9104e14e74b6d8c446ca3077b2952
I20250814 08:00:19.149557 20960 log_verifier.cc:177] Verified matching terms for 0 ops in tablet cbe9104e14e74b6d8c446ca3077b2952
I20250814 08:00:19.160691 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 27374
I20250814 08:00:19.213923 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 27528
I20250814 08:00:19.261603 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 27663
I20250814 08:00:19.292310 20960 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskqdzHdL/build/tsan/bin/kudu with pid 27303
2025-08-14T08:00:19Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (37826 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (37826 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (189055 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST