Note: This is test shard 6 of 8.
[==========] Running 9 tests from 5 test suites.
[----------] Global test environment set-up.
[----------] 5 tests from AdminCliTest
[ RUN ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250905 13:57:17.483659 21899 test_util.cc:276] Using random seed: 926755314
W20250905 13:57:18.629385 21899 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.099s user 0.466s sys 0.630s
W20250905 13:57:18.629814 21899 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.099s user 0.466s sys 0.630s
I20250905 13:57:18.632125 21899 ts_itest-base.cc:115] Starting cluster with:
I20250905 13:57:18.632341 21899 ts_itest-base.cc:116] --------------
I20250905 13:57:18.632519 21899 ts_itest-base.cc:117] 4 tablet servers
I20250905 13:57:18.632717 21899 ts_itest-base.cc:118] 3 replicas per TS
I20250905 13:57:18.632885 21899 ts_itest-base.cc:119] --------------
2025-09-05T13:57:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T13:57:18Z Disabled control of system clock
I20250905 13:57:18.672454 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:38711
--webserver_interface=127.21.98.254
--webserver_port=0
--builtin_ntp_servers=127.21.98.212:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:38711 with env {}
W20250905 13:57:18.964913 21913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:18.965401 21913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:18.965830 21913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:18.994661 21913 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:57:18.994947 21913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:18.995168 21913 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:57:18.995378 21913 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:57:19.027594 21913 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:38711
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:38711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:19.028973 21913 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:19.030679 21913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:19.040580 21919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:20.444947 21918 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 21913
W20250905 13:57:20.740072 21913 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.699s user 0.652s sys 1.047s
W20250905 13:57:20.740645 21913 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.700s user 0.652s sys 1.047s
W20250905 13:57:19.041265 21920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:20.742352 21921 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1699 milliseconds
I20250905 13:57:20.743209 21913 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250905 13:57:20.743252 21922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:57:20.746752 21913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:20.749118 21913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:20.750442 21913 hybrid_clock.cc:648] HybridClock initialized: now 1757080640750382 us; error 58 us; skew 500 ppm
I20250905 13:57:20.751281 21913 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:20.756924 21913 webserver.cc:480] Webserver started at http://127.21.98.254:43553/ using document root <none> and password file <none>
I20250905 13:57:20.757781 21913 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:20.757972 21913 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:20.758358 21913 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:20.762604 21913 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "d9df4b6f8bce4e2486eed92925ef26d6"
format_stamp: "Formatted at 2025-09-05 13:57:20 on dist-test-slave-g001"
I20250905 13:57:20.763535 21913 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "d9df4b6f8bce4e2486eed92925ef26d6"
format_stamp: "Formatted at 2025-09-05 13:57:20 on dist-test-slave-g001"
I20250905 13:57:20.769989 21913 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20250905 13:57:20.775527 21929 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:20.776579 21913 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250905 13:57:20.776964 21913 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d9df4b6f8bce4e2486eed92925ef26d6"
format_stamp: "Formatted at 2025-09-05 13:57:20 on dist-test-slave-g001"
I20250905 13:57:20.777245 21913 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:20.833576 21913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:20.834785 21913 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:20.835165 21913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:20.901124 21913 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:38711
I20250905 13:57:20.901202 21980 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:38711 every 8 connection(s)
I20250905 13:57:20.903630 21913 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 13:57:20.908975 21981 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:20.912065 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 21913
I20250905 13:57:20.912614 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 13:57:20.928681 21981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6: Bootstrap starting.
I20250905 13:57:20.933585 21981 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:20.935103 21981 log.cc:826] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:20.940079 21981 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6: No bootstrap required, opened a new log
I20250905 13:57:20.958389 21981 raft_consensus.cc:357] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } }
I20250905 13:57:20.959054 21981 raft_consensus.cc:383] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:20.959250 21981 raft_consensus.cc:738] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9df4b6f8bce4e2486eed92925ef26d6, State: Initialized, Role: FOLLOWER
I20250905 13:57:20.959810 21981 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } }
I20250905 13:57:20.960263 21981 raft_consensus.cc:397] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:57:20.960464 21981 raft_consensus.cc:491] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:57:20.960711 21981 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:20.964637 21981 raft_consensus.cc:513] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } }
I20250905 13:57:20.965277 21981 leader_election.cc:304] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9df4b6f8bce4e2486eed92925ef26d6; no voters:
I20250905 13:57:20.966812 21981 leader_election.cc:290] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:57:20.967464 21986 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:57:20.969473 21986 raft_consensus.cc:695] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 1 LEADER]: Becoming Leader. State: Replica: d9df4b6f8bce4e2486eed92925ef26d6, State: Running, Role: LEADER
I20250905 13:57:20.970170 21986 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } }
I20250905 13:57:20.971122 21981 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:57:20.978488 21988 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9df4b6f8bce4e2486eed92925ef26d6. Latest consensus state: current_term: 1 leader_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } } }
I20250905 13:57:20.978636 21987 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } } }
I20250905 13:57:20.979188 21988 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [sys.catalog]: This master's current role is: LEADER
I20250905 13:57:20.979190 21987 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [sys.catalog]: This master's current role is: LEADER
I20250905 13:57:20.984828 21993 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:57:20.995090 21993 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:57:21.012396 21993 catalog_manager.cc:1349] Generated new cluster ID: d329d09885b04cb2be73d4d7706f60fb
I20250905 13:57:21.012669 21993 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:57:21.028625 21993 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 13:57:21.029946 21993 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:57:21.043388 21993 catalog_manager.cc:5955] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6: Generated new TSK 0
I20250905 13:57:21.044057 21993 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 13:57:21.052176 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:0
--local_ip_for_outbound_sockets=127.21.98.193
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:38711
--builtin_ntp_servers=127.21.98.212:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250905 13:57:21.333580 22005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:21.334041 22005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:21.334484 22005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:21.364581 22005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:21.365432 22005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 13:57:21.400132 22005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:38711
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:21.401868 22005 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:21.404083 22005 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:21.416666 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:21.419441 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:21.422641 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:22.872318 22013 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1451 milliseconds
I20250905 13:57:22.872408 22005 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:22.873466 22005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:22.875948 22005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:22.877363 22005 hybrid_clock.cc:648] HybridClock initialized: now 1757080642877336 us; error 48 us; skew 500 ppm
I20250905 13:57:22.878096 22005 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:22.884114 22005 webserver.cc:480] Webserver started at http://127.21.98.193:44431/ using document root <none> and password file <none>
I20250905 13:57:22.884954 22005 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:22.885135 22005 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:22.885531 22005 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:22.889573 22005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e7123a5f681943ae91c7e39c757ca976"
format_stamp: "Formatted at 2025-09-05 13:57:22 on dist-test-slave-g001"
I20250905 13:57:22.890631 22005 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e7123a5f681943ae91c7e39c757ca976"
format_stamp: "Formatted at 2025-09-05 13:57:22 on dist-test-slave-g001"
I20250905 13:57:22.897092 22005 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250905 13:57:22.902051 22021 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:22.902878 22005 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 13:57:22.903194 22005 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e7123a5f681943ae91c7e39c757ca976"
format_stamp: "Formatted at 2025-09-05 13:57:22 on dist-test-slave-g001"
I20250905 13:57:22.903481 22005 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:22.947245 22005 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:22.948516 22005 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:22.948882 22005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:22.951150 22005 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:57:22.954753 22005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:57:22.954957 22005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250905 13:57:22.955153 22005 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:57:22.955279 22005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:23.080533 22005 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:41891
I20250905 13:57:23.080621 22133 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:41891 every 8 connection(s)
I20250905 13:57:23.083091 22005 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 13:57:23.086998 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 22005
I20250905 13:57:23.087417 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 13:57:23.093508 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.194:0
--local_ip_for_outbound_sockets=127.21.98.194
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:38711
--builtin_ntp_servers=127.21.98.212:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:57:23.104092 22134 heartbeater.cc:344] Connected to a master server at 127.21.98.254:38711
I20250905 13:57:23.104418 22134 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:23.105294 22134 heartbeater.cc:507] Master 127.21.98.254:38711 requested a full tablet report, sending...
I20250905 13:57:23.107815 21946 ts_manager.cc:194] Registered new tserver with Master: e7123a5f681943ae91c7e39c757ca976 (127.21.98.193:41891)
I20250905 13:57:23.110487 21946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:54297
W20250905 13:57:23.379065 22138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:23.379509 22138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:23.379894 22138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:23.411021 22138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:23.411794 22138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.194
I20250905 13:57:23.443662 22138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:38711
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.194
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:23.444872 22138 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:23.446434 22138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:23.456794 22144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:23.788488 21977 debug-util.cc:398] Leaking SignalData structure 0x7b080006f660 after lost signal to thread 21914
I20250905 13:57:24.113970 22134 heartbeater.cc:499] Master 127.21.98.254:38711 was elected leader, sending a full tablet report...
W20250905 13:57:23.458859 22145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:24.858440 22147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:24.860692 22138 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.403s user 0.462s sys 0.911s
W20250905 13:57:24.861055 22138 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.403s user 0.462s sys 0.911s
W20250905 13:57:24.867393 22146 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1399 milliseconds
I20250905 13:57:24.867419 22138 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:24.868959 22138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:24.871647 22138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:24.873140 22138 hybrid_clock.cc:648] HybridClock initialized: now 1757080644873059 us; error 78 us; skew 500 ppm
I20250905 13:57:24.874260 22138 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:24.882826 22138 webserver.cc:480] Webserver started at http://127.21.98.194:44697/ using document root <none> and password file <none>
I20250905 13:57:24.883976 22138 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:24.884306 22138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:24.884893 22138 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:24.891952 22138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "f3d86d97efe04b5d84598d40d7d1e891"
format_stamp: "Formatted at 2025-09-05 13:57:24 on dist-test-slave-g001"
I20250905 13:57:24.893584 22138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "f3d86d97efe04b5d84598d40d7d1e891"
format_stamp: "Formatted at 2025-09-05 13:57:24 on dist-test-slave-g001"
I20250905 13:57:24.903030 22138 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20250905 13:57:24.910413 22154 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:24.911715 22138 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20250905 13:57:24.912144 22138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "f3d86d97efe04b5d84598d40d7d1e891"
format_stamp: "Formatted at 2025-09-05 13:57:24 on dist-test-slave-g001"
I20250905 13:57:24.912554 22138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:24.988495 22138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:24.989939 22138 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:24.990274 22138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:24.993211 22138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:57:24.997187 22138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:57:24.997356 22138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:24.997545 22138 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:57:24.997656 22138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:25.163318 22138 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.194:41405
I20250905 13:57:25.163491 22266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.194:41405 every 8 connection(s)
I20250905 13:57:25.165759 22138 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 13:57:25.174212 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 22138
I20250905 13:57:25.174677 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 13:57:25.182463 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.195:0
--local_ip_for_outbound_sockets=127.21.98.195
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:38711
--builtin_ntp_servers=127.21.98.212:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:57:25.186564 22267 heartbeater.cc:344] Connected to a master server at 127.21.98.254:38711
I20250905 13:57:25.187004 22267 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:25.188030 22267 heartbeater.cc:507] Master 127.21.98.254:38711 requested a full tablet report, sending...
I20250905 13:57:25.189874 21946 ts_manager.cc:194] Registered new tserver with Master: f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194:41405)
I20250905 13:57:25.191035 21946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.194:50489
W20250905 13:57:25.463779 22271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:25.464207 22271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:25.464648 22271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:25.493120 22271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:25.493901 22271 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.195
I20250905 13:57:25.525018 22271 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:38711
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.195
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:25.526211 22271 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:25.527745 22271 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:25.540089 22277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:57:26.194597 22267 heartbeater.cc:499] Master 127.21.98.254:38711 was elected leader, sending a full tablet report...
W20250905 13:57:25.544523 22280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:25.542375 22278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:26.691414 22279 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1144 milliseconds
I20250905 13:57:26.691563 22271 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:26.692808 22271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:26.694710 22271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:26.696019 22271 hybrid_clock.cc:648] HybridClock initialized: now 1757080646696003 us; error 45 us; skew 500 ppm
I20250905 13:57:26.696853 22271 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:26.702740 22271 webserver.cc:480] Webserver started at http://127.21.98.195:42319/ using document root <none> and password file <none>
I20250905 13:57:26.703552 22271 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:26.703732 22271 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:26.704134 22271 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:26.708177 22271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "8cff93f53bd04e1799b3c1da27bb13cb"
format_stamp: "Formatted at 2025-09-05 13:57:26 on dist-test-slave-g001"
I20250905 13:57:26.709117 22271 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "8cff93f53bd04e1799b3c1da27bb13cb"
format_stamp: "Formatted at 2025-09-05 13:57:26 on dist-test-slave-g001"
I20250905 13:57:26.715677 22271 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20250905 13:57:26.720566 22287 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:26.721493 22271 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20250905 13:57:26.721801 22271 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "8cff93f53bd04e1799b3c1da27bb13cb"
format_stamp: "Formatted at 2025-09-05 13:57:26 on dist-test-slave-g001"
I20250905 13:57:26.722106 22271 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:26.784672 22271 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:26.785910 22271 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:26.786264 22271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:26.788411 22271 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:57:26.792150 22271 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:57:26.792377 22271 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:26.792595 22271 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:57:26.792730 22271 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:26.921132 22271 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.195:38797
I20250905 13:57:26.921234 22399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.195:38797 every 8 connection(s)
I20250905 13:57:26.923823 22271 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 13:57:26.931277 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 22271
I20250905 13:57:26.931794 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 13:57:26.938410 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.196:0
--local_ip_for_outbound_sockets=127.21.98.196
--webserver_interface=127.21.98.196
--webserver_port=0
--tserver_master_addrs=127.21.98.254:38711
--builtin_ntp_servers=127.21.98.212:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:57:26.950016 22400 heartbeater.cc:344] Connected to a master server at 127.21.98.254:38711
I20250905 13:57:26.950511 22400 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:26.951952 22400 heartbeater.cc:507] Master 127.21.98.254:38711 requested a full tablet report, sending...
I20250905 13:57:26.954329 21946 ts_manager.cc:194] Registered new tserver with Master: 8cff93f53bd04e1799b3c1da27bb13cb (127.21.98.195:38797)
I20250905 13:57:26.956172 21946 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.195:55089
W20250905 13:57:27.239217 22404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:27.239656 22404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:27.240037 22404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:27.269410 22404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:27.270156 22404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.196
I20250905 13:57:27.303812 22404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.98.196
--webserver_port=0
--tserver_master_addrs=127.21.98.254:38711
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.196
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:27.305052 22404 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:27.306622 22404 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:27.317287 22410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:57:27.959808 22400 heartbeater.cc:499] Master 127.21.98.254:38711 was elected leader, sending a full tablet report...
W20250905 13:57:27.321961 22413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:27.318989 22411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:28.408454 22412 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250905 13:57:28.408674 22404 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:28.411959 22404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:28.414412 22404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:28.415815 22404 hybrid_clock.cc:648] HybridClock initialized: now 1757080648415764 us; error 50 us; skew 500 ppm
I20250905 13:57:28.416574 22404 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:28.422492 22404 webserver.cc:480] Webserver started at http://127.21.98.196:40607/ using document root <none> and password file <none>
I20250905 13:57:28.423305 22404 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:28.423491 22404 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:28.423909 22404 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:28.427779 22404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "76519eb6c9fc41058582e9ac253ec248"
format_stamp: "Formatted at 2025-09-05 13:57:28 on dist-test-slave-g001"
I20250905 13:57:28.428706 22404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "76519eb6c9fc41058582e9ac253ec248"
format_stamp: "Formatted at 2025-09-05 13:57:28 on dist-test-slave-g001"
I20250905 13:57:28.435303 22404 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250905 13:57:28.440747 22421 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:28.441741 22404 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250905 13:57:28.442013 22404 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "76519eb6c9fc41058582e9ac253ec248"
format_stamp: "Formatted at 2025-09-05 13:57:28 on dist-test-slave-g001"
I20250905 13:57:28.442289 22404 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:28.489214 22404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:28.490473 22404 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:28.490859 22404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:28.493599 22404 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:57:28.497612 22404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:57:28.497794 22404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:28.497977 22404 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:57:28.498093 22404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:28.624750 22404 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.196:36233
I20250905 13:57:28.624845 22533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.196:36233 every 8 connection(s)
I20250905 13:57:28.627313 22404 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250905 13:57:28.629993 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 22404
I20250905 13:57:28.630715 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250905 13:57:28.647046 22534 heartbeater.cc:344] Connected to a master server at 127.21.98.254:38711
I20250905 13:57:28.647403 22534 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:28.648284 22534 heartbeater.cc:507] Master 127.21.98.254:38711 requested a full tablet report, sending...
I20250905 13:57:28.650219 21945 ts_manager.cc:194] Registered new tserver with Master: 76519eb6c9fc41058582e9ac253ec248 (127.21.98.196:36233)
I20250905 13:57:28.651893 21945 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.196:46837
I20250905 13:57:28.652295 21899 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250905 13:57:28.689303 21945 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59402:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250905 13:57:28.754395 22202 tablet_service.cc:1468] Processing CreateTablet for tablet a3968b2ae6a945c893695492d5a7dd6d (DEFAULT_TABLE table=TestTable [id=d88209b12aeb45c7834074d426cfa9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:57:28.756547 22202 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3968b2ae6a945c893695492d5a7dd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:28.757086 22335 tablet_service.cc:1468] Processing CreateTablet for tablet a3968b2ae6a945c893695492d5a7dd6d (DEFAULT_TABLE table=TestTable [id=d88209b12aeb45c7834074d426cfa9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:57:28.758783 22335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3968b2ae6a945c893695492d5a7dd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:28.758548 22469 tablet_service.cc:1468] Processing CreateTablet for tablet a3968b2ae6a945c893695492d5a7dd6d (DEFAULT_TABLE table=TestTable [id=d88209b12aeb45c7834074d426cfa9f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:57:28.759934 22469 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3968b2ae6a945c893695492d5a7dd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:28.779448 22553 tablet_bootstrap.cc:492] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248: Bootstrap starting.
I20250905 13:57:28.783281 22554 tablet_bootstrap.cc:492] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891: Bootstrap starting.
I20250905 13:57:28.785275 22555 tablet_bootstrap.cc:492] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb: Bootstrap starting.
I20250905 13:57:28.789108 22553 tablet_bootstrap.cc:654] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:28.789938 22554 tablet_bootstrap.cc:654] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:28.791635 22554 log.cc:826] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:28.791626 22553 log.cc:826] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:28.793051 22555 tablet_bootstrap.cc:654] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:28.795724 22555 log.cc:826] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:28.796909 22553 tablet_bootstrap.cc:492] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248: No bootstrap required, opened a new log
I20250905 13:57:28.797325 22553 ts_tablet_manager.cc:1397] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.007s
I20250905 13:57:28.797367 22554 tablet_bootstrap.cc:492] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891: No bootstrap required, opened a new log
I20250905 13:57:28.798010 22554 ts_tablet_manager.cc:1397] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20250905 13:57:28.801421 22555 tablet_bootstrap.cc:492] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb: No bootstrap required, opened a new log
I20250905 13:57:28.801952 22555 ts_tablet_manager.cc:1397] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20250905 13:57:28.816465 22554 raft_consensus.cc:357] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:28.817370 22554 raft_consensus.cc:383] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:28.817672 22554 raft_consensus.cc:738] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3d86d97efe04b5d84598d40d7d1e891, State: Initialized, Role: FOLLOWER
I20250905 13:57:28.818694 22554 consensus_queue.cc:260] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:28.823779 22553 raft_consensus.cc:357] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:28.824651 22553 raft_consensus.cc:383] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:28.824928 22553 raft_consensus.cc:738] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76519eb6c9fc41058582e9ac253ec248, State: Initialized, Role: FOLLOWER
I20250905 13:57:28.825583 22554 ts_tablet_manager.cc:1428] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891: Time spent starting tablet: real 0.027s user 0.013s sys 0.012s
I20250905 13:57:28.825978 22553 consensus_queue.cc:260] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:28.827531 22555 raft_consensus.cc:357] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:28.828465 22555 raft_consensus.cc:383] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:28.828773 22555 raft_consensus.cc:738] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cff93f53bd04e1799b3c1da27bb13cb, State: Initialized, Role: FOLLOWER
I20250905 13:57:28.828931 22534 heartbeater.cc:499] Master 127.21.98.254:38711 was elected leader, sending a full tablet report...
I20250905 13:57:28.829756 22555 consensus_queue.cc:260] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:28.830294 22553 ts_tablet_manager.cc:1428] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248: Time spent starting tablet: real 0.033s user 0.027s sys 0.003s
I20250905 13:57:28.833576 22555 ts_tablet_manager.cc:1428] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb: Time spent starting tablet: real 0.031s user 0.024s sys 0.006s
W20250905 13:57:28.881199 22535 tablet.cc:2378] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 13:57:28.924069 22268 tablet.cc:2378] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 13:57:28.929670 22401 tablet.cc:2378] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:57:29.006237 22559 raft_consensus.cc:491] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:57:29.006685 22559 raft_consensus.cc:513] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:29.009344 22559 leader_election.cc:290] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8cff93f53bd04e1799b3c1da27bb13cb (127.21.98.195:38797), 76519eb6c9fc41058582e9ac253ec248 (127.21.98.196:36233)
I20250905 13:57:29.019886 22355 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3968b2ae6a945c893695492d5a7dd6d" candidate_uuid: "f3d86d97efe04b5d84598d40d7d1e891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" is_pre_election: true
I20250905 13:57:29.020515 22355 raft_consensus.cc:2466] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3d86d97efe04b5d84598d40d7d1e891 in term 0.
I20250905 13:57:29.021070 22489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3968b2ae6a945c893695492d5a7dd6d" candidate_uuid: "f3d86d97efe04b5d84598d40d7d1e891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76519eb6c9fc41058582e9ac253ec248" is_pre_election: true
I20250905 13:57:29.021549 22156 leader_election.cc:304] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cff93f53bd04e1799b3c1da27bb13cb, f3d86d97efe04b5d84598d40d7d1e891; no voters:
I20250905 13:57:29.021680 22489 raft_consensus.cc:2466] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3d86d97efe04b5d84598d40d7d1e891 in term 0.
I20250905 13:57:29.022233 22559 raft_consensus.cc:2802] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 13:57:29.022562 22559 raft_consensus.cc:491] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 13:57:29.022825 22559 raft_consensus.cc:3058] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:29.026705 22559 raft_consensus.cc:513] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:29.027834 22559 leader_election.cc:290] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [CANDIDATE]: Term 1 election: Requested vote from peers 8cff93f53bd04e1799b3c1da27bb13cb (127.21.98.195:38797), 76519eb6c9fc41058582e9ac253ec248 (127.21.98.196:36233)
I20250905 13:57:29.028466 22355 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3968b2ae6a945c893695492d5a7dd6d" candidate_uuid: "f3d86d97efe04b5d84598d40d7d1e891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8cff93f53bd04e1799b3c1da27bb13cb"
I20250905 13:57:29.028610 22489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3968b2ae6a945c893695492d5a7dd6d" candidate_uuid: "f3d86d97efe04b5d84598d40d7d1e891" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76519eb6c9fc41058582e9ac253ec248"
I20250905 13:57:29.028905 22355 raft_consensus.cc:3058] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:29.029009 22489 raft_consensus.cc:3058] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:29.035176 22489 raft_consensus.cc:2466] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3d86d97efe04b5d84598d40d7d1e891 in term 1.
I20250905 13:57:29.035176 22355 raft_consensus.cc:2466] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3d86d97efe04b5d84598d40d7d1e891 in term 1.
I20250905 13:57:29.036091 22156 leader_election.cc:304] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cff93f53bd04e1799b3c1da27bb13cb, f3d86d97efe04b5d84598d40d7d1e891; no voters:
I20250905 13:57:29.036643 22559 raft_consensus.cc:2802] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:57:29.038091 22559 raft_consensus.cc:695] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [term 1 LEADER]: Becoming Leader. State: Replica: f3d86d97efe04b5d84598d40d7d1e891, State: Running, Role: LEADER
I20250905 13:57:29.038851 22559 consensus_queue.cc:237] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:29.047395 21946 catalog_manager.cc:5582] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 reported cstate change: term changed from 0 to 1, leader changed from <none> to f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194). New cstate: current_term: 1 leader_uuid: "f3d86d97efe04b5d84598d40d7d1e891" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } health_report { overall_health: UNKNOWN } } }
I20250905 13:57:29.120064 21899 external_mini_cluster.cc:949] 4 TS(s) registered with all masters
I20250905 13:57:29.123940 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver f3d86d97efe04b5d84598d40d7d1e891 to finish bootstrapping
I20250905 13:57:29.137660 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 8cff93f53bd04e1799b3c1da27bb13cb to finish bootstrapping
I20250905 13:57:29.148649 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 76519eb6c9fc41058582e9ac253ec248 to finish bootstrapping
I20250905 13:57:29.159430 21899 kudu-admin-test.cc:709] Waiting for Master to see the current replicas...
I20250905 13:57:29.163399 21899 kudu-admin-test.cc:716] Tablet locations:
tablet_locations {
tablet_id: "a3968b2ae6a945c893695492d5a7dd6d"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891"
rpc_addresses {
host: "127.21.98.194"
port: 41405
}
}
ts_infos {
permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb"
rpc_addresses {
host: "127.21.98.195"
port: 38797
}
}
ts_infos {
permanent_uuid: "76519eb6c9fc41058582e9ac253ec248"
rpc_addresses {
host: "127.21.98.196"
port: 36233
}
}
I20250905 13:57:29.566648 22569 consensus_queue.cc:1035] T a3968b2ae6a945c893695492d5a7dd6d P f3d86d97efe04b5d84598d40d7d1e891 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 13:57:29.582191 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 22138
W20250905 13:57:29.607515 22423 connection.cc:537] server connection from 127.21.98.194:60447 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 13:57:29.608150 21932 connection.cc:537] server connection from 127.21.98.194:50489 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 13:57:29.608949 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 21913
I20250905 13:57:29.635555 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:38711
--webserver_interface=127.21.98.254
--webserver_port=43553
--builtin_ntp_servers=127.21.98.212:35997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:38711 with env {}
W20250905 13:57:29.845299 22400 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:38711 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:38711: connect: Connection refused (error 111)
W20250905 13:57:29.918947 22576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:29.919692 22576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:29.920310 22576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:29.959406 22576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:57:29.959664 22576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:29.959867 22576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:57:29.960070 22576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:57:29.993340 22576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:35997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:38711
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:38711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=43553
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:29.994469 22576 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:29.996112 22576 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:30.005649 22583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:30.159163 22134 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:38711 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:38711: connect: Connection refused (error 111)
W20250905 13:57:30.609473 22534 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:38711 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:38711: connect: Connection refused (error 111)
I20250905 13:57:31.108072 22592 raft_consensus.cc:491] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f3d86d97efe04b5d84598d40d7d1e891)
I20250905 13:57:31.108752 22592 raft_consensus.cc:513] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:31.117980 22592 leader_election.cc:290] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194:41405), 8cff93f53bd04e1799b3c1da27bb13cb (127.21.98.195:38797)
W20250905 13:57:31.129638 22423 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.98.194:41405: connect: Connection refused (error 111)
W20250905 13:57:31.143251 22423 leader_election.cc:336] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194:41405): Network error: Client connection negotiation failed: client connection to 127.21.98.194:41405: connect: Connection refused (error 111)
I20250905 13:57:31.144665 22355 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3968b2ae6a945c893695492d5a7dd6d" candidate_uuid: "76519eb6c9fc41058582e9ac253ec248" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" is_pre_election: true
I20250905 13:57:31.145413 22355 raft_consensus.cc:2466] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 76519eb6c9fc41058582e9ac253ec248 in term 1.
I20250905 13:57:31.146958 22423 leader_election.cc:304] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 76519eb6c9fc41058582e9ac253ec248, 8cff93f53bd04e1799b3c1da27bb13cb; no voters: f3d86d97efe04b5d84598d40d7d1e891
I20250905 13:57:31.147903 22592 raft_consensus.cc:2802] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 13:57:31.148254 22592 raft_consensus.cc:491] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 1 FOLLOWER]: Starting leader election (detected failure of leader f3d86d97efe04b5d84598d40d7d1e891)
I20250905 13:57:31.148599 22592 raft_consensus.cc:3058] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 1 FOLLOWER]: Advancing to term 2
W20250905 13:57:31.153340 22585 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1144 milliseconds
W20250905 13:57:30.007592 22584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:31.156181 22586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:57:31.156184 22592 raft_consensus.cc:513] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:31.157727 22592 leader_election.cc:290] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [CANDIDATE]: Term 2 election: Requested vote from peers f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194:41405), 8cff93f53bd04e1799b3c1da27bb13cb (127.21.98.195:38797)
W20250905 13:57:31.158982 22576 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.152s user 0.004s sys 0.004s
I20250905 13:57:31.158993 22355 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3968b2ae6a945c893695492d5a7dd6d" candidate_uuid: "76519eb6c9fc41058582e9ac253ec248" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8cff93f53bd04e1799b3c1da27bb13cb"
W20250905 13:57:31.159250 22576 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.152s user 0.004s sys 0.004s
I20250905 13:57:31.159458 22576 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:31.159530 22355 raft_consensus.cc:3058] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 1 FOLLOWER]: Advancing to term 2
I20250905 13:57:31.162772 22576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:31.166146 22576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:31.166329 22355 raft_consensus.cc:2466] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 76519eb6c9fc41058582e9ac253ec248 in term 2.
I20250905 13:57:31.167191 22423 leader_election.cc:304] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 76519eb6c9fc41058582e9ac253ec248, 8cff93f53bd04e1799b3c1da27bb13cb; no voters:
I20250905 13:57:31.167819 22592 raft_consensus.cc:2802] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 13:57:31.168172 22576 hybrid_clock.cc:648] HybridClock initialized: now 1757080651168127 us; error 34 us; skew 500 ppm
W20250905 13:57:31.168889 22423 leader_election.cc:336] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194:41405): Network error: Client connection negotiation failed: client connection to 127.21.98.194:41405: connect: Connection refused (error 111)
I20250905 13:57:31.169698 22576 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:31.171339 22592 raft_consensus.cc:695] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 2 LEADER]: Becoming Leader. State: Replica: 76519eb6c9fc41058582e9ac253ec248, State: Running, Role: LEADER
I20250905 13:57:31.172375 22592 consensus_queue.cc:237] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:31.178867 22576 webserver.cc:480] Webserver started at http://127.21.98.254:43553/ using document root <none> and password file <none>
I20250905 13:57:31.179723 22576 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:31.179914 22576 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:31.187601 22576 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 13:57:31.192291 22604 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:31.193336 22576 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250905 13:57:31.193635 22576 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "d9df4b6f8bce4e2486eed92925ef26d6"
format_stamp: "Formatted at 2025-09-05 13:57:20 on dist-test-slave-g001"
I20250905 13:57:31.195490 22576 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:31.273102 22576 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:31.274446 22576 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:31.274869 22576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:31.347338 22576 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:38711
I20250905 13:57:31.347389 22656 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:38711 every 8 connection(s)
I20250905 13:57:31.349886 22576 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 13:57:31.352391 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 22576
I20250905 13:57:31.353084 21899 kudu-admin-test.cc:735] Forcing unsafe config change on tserver 8cff93f53bd04e1799b3c1da27bb13cb
I20250905 13:57:31.364799 22657 sys_catalog.cc:263] Verifying existing consensus state
I20250905 13:57:31.369272 22657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6: Bootstrap starting.
I20250905 13:57:31.407197 22657 log.cc:826] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:31.427800 22657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=5 ignored=0} mutations{seen=2 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:57:31.428452 22657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6: Bootstrap complete.
I20250905 13:57:31.448635 22657 raft_consensus.cc:357] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } }
I20250905 13:57:31.450371 22657 raft_consensus.cc:738] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9df4b6f8bce4e2486eed92925ef26d6, State: Initialized, Role: FOLLOWER
I20250905 13:57:31.451114 22657 consensus_queue.cc:260] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } }
I20250905 13:57:31.451571 22657 raft_consensus.cc:397] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:57:31.451830 22657 raft_consensus.cc:491] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:57:31.452124 22657 raft_consensus.cc:3058] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 1 FOLLOWER]: Advancing to term 2
I20250905 13:57:31.457243 22657 raft_consensus.cc:513] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } }
I20250905 13:57:31.457937 22657 leader_election.cc:304] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d9df4b6f8bce4e2486eed92925ef26d6; no voters:
I20250905 13:57:31.460069 22657 leader_election.cc:290] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 13:57:31.460491 22661 raft_consensus.cc:2802] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 13:57:31.463630 22661 raft_consensus.cc:695] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [term 2 LEADER]: Becoming Leader. State: Replica: d9df4b6f8bce4e2486eed92925ef26d6, State: Running, Role: LEADER
I20250905 13:57:31.464440 22661 consensus_queue.cc:237] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } }
I20250905 13:57:31.465092 22657 sys_catalog.cc:564] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:57:31.471851 22662 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } } }
I20250905 13:57:31.472853 22662 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [sys.catalog]: This master's current role is: LEADER
I20250905 13:57:31.475198 22663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d9df4b6f8bce4e2486eed92925ef26d6. Latest consensus state: current_term: 2 leader_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d9df4b6f8bce4e2486eed92925ef26d6" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 38711 } } }
I20250905 13:57:31.475862 22663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6 [sys.catalog]: This master's current role is: LEADER
I20250905 13:57:31.478863 22667 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:57:31.490679 22667 catalog_manager.cc:671] Loaded metadata for table TestTable [id=d88209b12aeb45c7834074d426cfa9f8]
I20250905 13:57:31.497875 22667 tablet_loader.cc:96] loaded metadata for tablet a3968b2ae6a945c893695492d5a7dd6d (table TestTable [id=d88209b12aeb45c7834074d426cfa9f8])
I20250905 13:57:31.499099 22667 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:57:31.503345 22667 catalog_manager.cc:1261] Loaded cluster ID: d329d09885b04cb2be73d4d7706f60fb
I20250905 13:57:31.503645 22667 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:57:31.510885 22667 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:57:31.515223 22667 catalog_manager.cc:5966] T 00000000000000000000000000000000 P d9df4b6f8bce4e2486eed92925ef26d6: Loaded TSK: 0
I20250905 13:57:31.516768 22667 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250905 13:57:31.683719 22659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:31.684227 22659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:31.713403 22659 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250905 13:57:31.814085 22355 raft_consensus.cc:1273] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 2 FOLLOWER]: Refusing update from remote peer 76519eb6c9fc41058582e9ac253ec248: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20250905 13:57:31.815340 22423 consensus_peers.cc:489] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 -> Peer f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194:41405): Couldn't send request to peer f3d86d97efe04b5d84598d40d7d1e891. Status: Network error: Client connection negotiation failed: client connection to 127.21.98.194:41405: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 13:57:31.816368 22688 consensus_queue.cc:1035] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 13:57:31.866129 22400 heartbeater.cc:344] Connected to a master server at 127.21.98.254:38711
I20250905 13:57:31.874250 22621 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" instance_seqno: 1757080646890520) as {username='slave'} at 127.21.98.195:54293; Asking this server to re-register.
I20250905 13:57:31.876001 22400 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:31.876601 22400 heartbeater.cc:507] Master 127.21.98.254:38711 requested a full tablet report, sending...
I20250905 13:57:31.879948 22621 ts_manager.cc:194] Registered new tserver with Master: 8cff93f53bd04e1799b3c1da27bb13cb (127.21.98.195:38797)
I20250905 13:57:31.885363 22621 catalog_manager.cc:5582] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb reported cstate change: term changed from 1 to 2, leader changed from f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194) to 76519eb6c9fc41058582e9ac253ec248 (127.21.98.196). New cstate: current_term: 2 leader_uuid: "76519eb6c9fc41058582e9ac253ec248" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } }
I20250905 13:57:31.911530 22534 heartbeater.cc:344] Connected to a master server at 127.21.98.254:38711
I20250905 13:57:31.915313 22620 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" instance_seqno: 1757080648596489) as {username='slave'} at 127.21.98.196:37123; Asking this server to re-register.
I20250905 13:57:31.917452 22534 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:31.918192 22534 heartbeater.cc:507] Master 127.21.98.254:38711 requested a full tablet report, sending...
I20250905 13:57:31.928656 22620 ts_manager.cc:194] Registered new tserver with Master: 76519eb6c9fc41058582e9ac253ec248 (127.21.98.196:36233)
I20250905 13:57:32.186112 22134 heartbeater.cc:344] Connected to a master server at 127.21.98.254:38711
I20250905 13:57:32.189005 22620 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" instance_seqno: 1757080643051665) as {username='slave'} at 127.21.98.193:36307; Asking this server to re-register.
I20250905 13:57:32.190578 22134 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:32.191499 22134 heartbeater.cc:507] Master 127.21.98.254:38711 requested a full tablet report, sending...
I20250905 13:57:32.194028 22620 ts_manager.cc:194] Registered new tserver with Master: e7123a5f681943ae91c7e39c757ca976 (127.21.98.193:41891)
W20250905 13:57:33.109339 22659 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.354s user 0.537s sys 0.810s
W20250905 13:57:33.109694 22659 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.355s user 0.537s sys 0.810s
I20250905 13:57:33.177052 22355 tablet_service.cc:1905] Received UnsafeChangeConfig RPC: dest_uuid: "8cff93f53bd04e1799b3c1da27bb13cb"
tablet_id: "a3968b2ae6a945c893695492d5a7dd6d"
caller_id: "kudu-tools"
new_config {
peers {
permanent_uuid: "76519eb6c9fc41058582e9ac253ec248"
}
peers {
permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb"
}
}
from {username='slave'} at 127.0.0.1:45940
W20250905 13:57:33.178727 22355 raft_consensus.cc:2216] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 2 FOLLOWER]: PROCEEDING WITH UNSAFE CONFIG CHANGE ON THIS SERVER, COMMITTED CONFIG: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }NEW CONFIG: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } unsafe_config_change: true
I20250905 13:57:33.180127 22355 raft_consensus.cc:3058] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 2 FOLLOWER]: Advancing to term 3
I20250905 13:57:33.436775 22355 raft_consensus.cc:1238] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 3 FOLLOWER]: Rejecting Update request from peer 76519eb6c9fc41058582e9ac253ec248 for earlier term 2. Current term is 3. Ops: []
I20250905 13:57:33.437794 22712 consensus_queue.cc:1046] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 }, Status: INVALID_TERM, Last received: 2.2, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 13:57:33.438912 22712 raft_consensus.cc:3053] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 2 LEADER]: Stepping down as leader of term 2
I20250905 13:57:33.439157 22712 raft_consensus.cc:738] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 76519eb6c9fc41058582e9ac253ec248, State: Running, Role: LEADER
I20250905 13:57:33.439692 22712 consensus_queue.cc:260] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } }
I20250905 13:57:33.440546 22712 raft_consensus.cc:3058] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 2 FOLLOWER]: Advancing to term 3
I20250905 13:57:34.690197 22716 raft_consensus.cc:491] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 3 FOLLOWER]: Starting pre-election (detected failure of leader kudu-tools)
I20250905 13:57:34.690558 22716 raft_consensus.cc:513] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 3 FOLLOWER]: Starting pre-election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } unsafe_config_change: true
I20250905 13:57:34.692087 22716 leader_election.cc:290] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 76519eb6c9fc41058582e9ac253ec248 (127.21.98.196:36233)
I20250905 13:57:34.700520 22489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3968b2ae6a945c893695492d5a7dd6d" candidate_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "76519eb6c9fc41058582e9ac253ec248" is_pre_election: true
I20250905 13:57:34.700943 22489 raft_consensus.cc:2466] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cff93f53bd04e1799b3c1da27bb13cb in term 3.
I20250905 13:57:34.701862 22289 leader_election.cc:304] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76519eb6c9fc41058582e9ac253ec248, 8cff93f53bd04e1799b3c1da27bb13cb; no voters:
I20250905 13:57:34.702466 22716 raft_consensus.cc:2802] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 13:57:34.702697 22716 raft_consensus.cc:491] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 3 FOLLOWER]: Starting leader election (detected failure of leader kudu-tools)
I20250905 13:57:34.702991 22716 raft_consensus.cc:3058] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 3 FOLLOWER]: Advancing to term 4
I20250905 13:57:34.707666 22716 raft_consensus.cc:513] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 4 FOLLOWER]: Starting leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } unsafe_config_change: true
I20250905 13:57:34.708555 22716 leader_election.cc:290] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [CANDIDATE]: Term 4 election: Requested vote from peers 76519eb6c9fc41058582e9ac253ec248 (127.21.98.196:36233)
I20250905 13:57:34.709419 22489 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3968b2ae6a945c893695492d5a7dd6d" candidate_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" candidate_term: 4 candidate_status { last_received { term: 3 index: 3 } } ignore_live_leader: false dest_uuid: "76519eb6c9fc41058582e9ac253ec248"
I20250905 13:57:34.709905 22489 raft_consensus.cc:3058] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 3 FOLLOWER]: Advancing to term 4
I20250905 13:57:34.715490 22489 raft_consensus.cc:2466] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cff93f53bd04e1799b3c1da27bb13cb in term 4.
I20250905 13:57:34.716339 22289 leader_election.cc:304] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 76519eb6c9fc41058582e9ac253ec248, 8cff93f53bd04e1799b3c1da27bb13cb; no voters:
I20250905 13:57:34.717381 22716 raft_consensus.cc:2802] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 4 FOLLOWER]: Leader election won for term 4
I20250905 13:57:34.718515 22716 raft_consensus.cc:695] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 4 LEADER]: Becoming Leader. State: Replica: 8cff93f53bd04e1799b3c1da27bb13cb, State: Running, Role: LEADER
I20250905 13:57:34.719842 22716 consensus_queue.cc:237] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 3.3, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } unsafe_config_change: true
I20250905 13:57:34.727167 22620 catalog_manager.cc:5582] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb reported cstate change: term changed from 2 to 4, leader changed from 76519eb6c9fc41058582e9ac253ec248 (127.21.98.196) to 8cff93f53bd04e1799b3c1da27bb13cb (127.21.98.195), now has a pending config: VOTER 8cff93f53bd04e1799b3c1da27bb13cb (127.21.98.195), VOTER 76519eb6c9fc41058582e9ac253ec248 (127.21.98.196). New cstate: current_term: 4 leader_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3d86d97efe04b5d84598d40d7d1e891" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41405 } } peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } health_report { overall_health: UNKNOWN } } } pending_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } unsafe_config_change: true }
I20250905 13:57:35.299796 22489 raft_consensus.cc:1273] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 4 FOLLOWER]: Refusing update from remote peer 8cff93f53bd04e1799b3c1da27bb13cb: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 4 index: 4. (index mismatch)
I20250905 13:57:35.300990 22725 consensus_queue.cc:1035] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20250905 13:57:35.308156 22728 raft_consensus.cc:2953] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 4 LEADER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } unsafe_config_change: true }
I20250905 13:57:35.308976 22489 raft_consensus.cc:2953] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 4 FOLLOWER]: Committing config change with OpId 3.3: config changed from index -1 to 3, VOTER f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } unsafe_config_change: true }
I20250905 13:57:35.317790 22620 catalog_manager.cc:5582] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb reported cstate change: config changed from index -1 to 3, VOTER f3d86d97efe04b5d84598d40d7d1e891 (127.21.98.194) evicted, no longer has a pending config: VOTER 8cff93f53bd04e1799b3c1da27bb13cb (127.21.98.195), VOTER 76519eb6c9fc41058582e9ac253ec248 (127.21.98.196). New cstate: current_term: 4 leader_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
W20250905 13:57:35.325131 22620 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet a3968b2ae6a945c893695492d5a7dd6d on TS f3d86d97efe04b5d84598d40d7d1e891: Not found: failed to reset TS proxy: Could not find TS for UUID f3d86d97efe04b5d84598d40d7d1e891
I20250905 13:57:35.338837 22355 consensus_queue.cc:237] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 4.4, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: true } } unsafe_config_change: true
I20250905 13:57:35.342463 22489 raft_consensus.cc:1273] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 4 FOLLOWER]: Refusing update from remote peer 8cff93f53bd04e1799b3c1da27bb13cb: Log matching property violated. Preceding OpId in replica: term: 4 index: 4. Preceding OpId from leader: term: 4 index: 5. (index mismatch)
I20250905 13:57:35.344033 22725 consensus_queue.cc:1035] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 13:57:35.348909 22728 raft_consensus.cc:2953] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 4 LEADER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER e7123a5f681943ae91c7e39c757ca976 (127.21.98.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: true } } unsafe_config_change: true }
I20250905 13:57:35.350314 22489 raft_consensus.cc:2953] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 4 FOLLOWER]: Committing config change with OpId 4.5: config changed from index 3 to 5, NON_VOTER e7123a5f681943ae91c7e39c757ca976 (127.21.98.193) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: true } } unsafe_config_change: true }
I20250905 13:57:35.356523 22606 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet a3968b2ae6a945c893695492d5a7dd6d with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250905 13:57:35.357723 22288 consensus_peers.cc:489] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb -> Peer e7123a5f681943ae91c7e39c757ca976 (127.21.98.193:41891): Couldn't send request to peer e7123a5f681943ae91c7e39c757ca976. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a3968b2ae6a945c893695492d5a7dd6d. This is attempt 1: this message will repeat every 5th retry.
I20250905 13:57:35.360028 22621 catalog_manager.cc:5582] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb reported cstate change: config changed from index 3 to 5, NON_VOTER e7123a5f681943ae91c7e39c757ca976 (127.21.98.193) added. New cstate: current_term: 4 leader_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: true } health_report { overall_health: UNKNOWN } } unsafe_config_change: true }
W20250905 13:57:35.378019 22606 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet a3968b2ae6a945c893695492d5a7dd6d on TS f3d86d97efe04b5d84598d40d7d1e891 failed: Not found: failed to reset TS proxy: Could not find TS for UUID f3d86d97efe04b5d84598d40d7d1e891
I20250905 13:57:35.939159 22739 ts_tablet_manager.cc:927] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: Initiating tablet copy from peer 8cff93f53bd04e1799b3c1da27bb13cb (127.21.98.195:38797)
I20250905 13:57:35.941267 22739 tablet_copy_client.cc:323] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: tablet copy: Beginning tablet copy session from remote peer at address 127.21.98.195:38797
I20250905 13:57:35.953513 22375 tablet_copy_service.cc:140] P 8cff93f53bd04e1799b3c1da27bb13cb: Received BeginTabletCopySession request for tablet a3968b2ae6a945c893695492d5a7dd6d from peer e7123a5f681943ae91c7e39c757ca976 ({username='slave'} at 127.21.98.193:42051)
I20250905 13:57:35.954169 22375 tablet_copy_service.cc:161] P 8cff93f53bd04e1799b3c1da27bb13cb: Beginning new tablet copy session on tablet a3968b2ae6a945c893695492d5a7dd6d from peer e7123a5f681943ae91c7e39c757ca976 at {username='slave'} at 127.21.98.193:42051: session id = e7123a5f681943ae91c7e39c757ca976-a3968b2ae6a945c893695492d5a7dd6d
I20250905 13:57:35.961153 22375 tablet_copy_source_session.cc:215] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 13:57:35.966419 22739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3968b2ae6a945c893695492d5a7dd6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:35.983680 22739 tablet_copy_client.cc:806] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: tablet copy: Starting download of 0 data blocks...
I20250905 13:57:35.984215 22739 tablet_copy_client.cc:670] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: tablet copy: Starting download of 1 WAL segments...
I20250905 13:57:35.987186 22739 tablet_copy_client.cc:538] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 13:57:35.992347 22739 tablet_bootstrap.cc:492] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: Bootstrap starting.
I20250905 13:57:36.004930 22739 log.cc:826] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:36.014420 22739 tablet_bootstrap.cc:492] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:57:36.015059 22739 tablet_bootstrap.cc:492] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: Bootstrap complete.
I20250905 13:57:36.015566 22739 ts_tablet_manager.cc:1397] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.009s
I20250905 13:57:36.030418 22739 raft_consensus.cc:357] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976 [term 4 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: true } } unsafe_config_change: true
I20250905 13:57:36.031199 22739 raft_consensus.cc:738] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976 [term 4 LEARNER]: Becoming Follower/Learner. State: Replica: e7123a5f681943ae91c7e39c757ca976, State: Initialized, Role: LEARNER
I20250905 13:57:36.031703 22739 consensus_queue.cc:260] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 4.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: true } } unsafe_config_change: true
I20250905 13:57:36.034543 22739 ts_tablet_manager.cc:1428] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976: Time spent starting tablet: real 0.019s user 0.010s sys 0.009s
I20250905 13:57:36.036041 22375 tablet_copy_service.cc:342] P 8cff93f53bd04e1799b3c1da27bb13cb: Request end of tablet copy session e7123a5f681943ae91c7e39c757ca976-a3968b2ae6a945c893695492d5a7dd6d received from {username='slave'} at 127.21.98.193:42051
I20250905 13:57:36.036499 22375 tablet_copy_service.cc:434] P 8cff93f53bd04e1799b3c1da27bb13cb: ending tablet copy session e7123a5f681943ae91c7e39c757ca976-a3968b2ae6a945c893695492d5a7dd6d on tablet a3968b2ae6a945c893695492d5a7dd6d with peer e7123a5f681943ae91c7e39c757ca976
I20250905 13:57:36.404466 22088 raft_consensus.cc:1215] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976 [term 4 LEARNER]: Deduplicated request from leader. Original: 4.4->[4.5-4.5] Dedup: 4.5->[]
W20250905 13:57:36.544080 22606 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet a3968b2ae6a945c893695492d5a7dd6d on TS f3d86d97efe04b5d84598d40d7d1e891 failed: Not found: failed to reset TS proxy: Could not find TS for UUID f3d86d97efe04b5d84598d40d7d1e891
I20250905 13:57:36.880674 22746 raft_consensus.cc:1062] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb: attempting to promote NON_VOTER e7123a5f681943ae91c7e39c757ca976 to VOTER
I20250905 13:57:36.882325 22746 consensus_queue.cc:237] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 4.5, Last appended by leader: 0, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: false } } unsafe_config_change: true
I20250905 13:57:36.887224 22489 raft_consensus.cc:1273] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 4 FOLLOWER]: Refusing update from remote peer 8cff93f53bd04e1799b3c1da27bb13cb: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 13:57:36.887318 22088 raft_consensus.cc:1273] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976 [term 4 LEARNER]: Refusing update from remote peer 8cff93f53bd04e1799b3c1da27bb13cb: Log matching property violated. Preceding OpId in replica: term: 4 index: 5. Preceding OpId from leader: term: 4 index: 6. (index mismatch)
I20250905 13:57:36.888507 22745 consensus_queue.cc:1035] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20250905 13:57:36.889473 22728 consensus_queue.cc:1035] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20250905 13:57:36.895680 22728 raft_consensus.cc:2953] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb [term 4 LEADER]: Committing config change with OpId 4.6: config changed from index 5 to 6, e7123a5f681943ae91c7e39c757ca976 (127.21.98.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: false } } unsafe_config_change: true }
I20250905 13:57:36.897071 22489 raft_consensus.cc:2953] T a3968b2ae6a945c893695492d5a7dd6d P 76519eb6c9fc41058582e9ac253ec248 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, e7123a5f681943ae91c7e39c757ca976 (127.21.98.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: false } } unsafe_config_change: true }
I20250905 13:57:36.899545 22088 raft_consensus.cc:2953] T a3968b2ae6a945c893695492d5a7dd6d P e7123a5f681943ae91c7e39c757ca976 [term 4 FOLLOWER]: Committing config change with OpId 4.6: config changed from index 5 to 6, e7123a5f681943ae91c7e39c757ca976 (127.21.98.193) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: false } } unsafe_config_change: true }
I20250905 13:57:36.904788 22620 catalog_manager.cc:5582] T a3968b2ae6a945c893695492d5a7dd6d P 8cff93f53bd04e1799b3c1da27bb13cb reported cstate change: config changed from index 5 to 6, e7123a5f681943ae91c7e39c757ca976 (127.21.98.193) changed from NON_VOTER to VOTER. New cstate: current_term: 4 leader_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 38797 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76519eb6c9fc41058582e9ac253ec248" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 36233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e7123a5f681943ae91c7e39c757ca976" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41891 } attrs { promote: false } health_report { overall_health: HEALTHY } } unsafe_config_change: true }
I20250905 13:57:36.979570 21899 kudu-admin-test.cc:751] Waiting for Master to see new config...
I20250905 13:57:36.990926 21899 kudu-admin-test.cc:756] Tablet locations:
tablet_locations {
tablet_id: "a3968b2ae6a945c893695492d5a7dd6d"
DEPRECATED_stale: false
partition {
partition_key_start: ""
partition_key_end: ""
}
interned_replicas {
ts_info_idx: 0
role: LEADER
}
interned_replicas {
ts_info_idx: 1
role: FOLLOWER
}
interned_replicas {
ts_info_idx: 2
role: FOLLOWER
}
}
ts_infos {
permanent_uuid: "8cff93f53bd04e1799b3c1da27bb13cb"
rpc_addresses {
host: "127.21.98.195"
port: 38797
}
}
ts_infos {
permanent_uuid: "76519eb6c9fc41058582e9ac253ec248"
rpc_addresses {
host: "127.21.98.196"
port: 36233
}
}
ts_infos {
permanent_uuid: "e7123a5f681943ae91c7e39c757ca976"
rpc_addresses {
host: "127.21.98.193"
port: 41891
}
}
I20250905 13:57:36.994128 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 22005
I20250905 13:57:37.017359 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 22271
I20250905 13:57:37.036891 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 22404
I20250905 13:57:37.060587 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 22576
2025-09-05T13:57:37Z chronyd exiting
[ OK ] AdminCliTest.TestUnsafeChangeConfigForConfigWithTwoNodes (19639 ms)
[ RUN ] AdminCliTest.TestGracefulSpecificLeaderStepDown
I20250905 13:57:37.120091 21899 test_util.cc:276] Using random seed: 946391879
I20250905 13:57:37.125576 21899 ts_itest-base.cc:115] Starting cluster with:
I20250905 13:57:37.125764 21899 ts_itest-base.cc:116] --------------
I20250905 13:57:37.125918 21899 ts_itest-base.cc:117] 3 tablet servers
I20250905 13:57:37.126057 21899 ts_itest-base.cc:118] 3 replicas per TS
I20250905 13:57:37.126192 21899 ts_itest-base.cc:119] --------------
2025-09-05T13:57:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T13:57:37Z Disabled control of system clock
I20250905 13:57:37.159137 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:35679
--webserver_interface=127.21.98.254
--webserver_port=0
--builtin_ntp_servers=127.21.98.212:43347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:35679
--catalog_manager_wait_for_new_tablets_to_elect_leader=false with env {}
W20250905 13:57:37.424342 22765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:37.424784 22765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:37.425123 22765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:37.452528 22765 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:57:37.452831 22765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:37.453147 22765 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:57:37.453354 22765 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:57:37.486196 22765 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--catalog_manager_wait_for_new_tablets_to_elect_leader=false
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:35679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:35679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:37.487468 22765 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:37.488875 22765 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:37.499686 22771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:37.500284 22772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:38.615250 22774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:38.619843 22773 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1118 milliseconds
I20250905 13:57:38.619948 22765 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:38.621129 22765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:38.623646 22765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:38.624974 22765 hybrid_clock.cc:648] HybridClock initialized: now 1757080658624927 us; error 40 us; skew 500 ppm
I20250905 13:57:38.625739 22765 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:38.631870 22765 webserver.cc:480] Webserver started at http://127.21.98.254:33571/ using document root <none> and password file <none>
I20250905 13:57:38.632936 22765 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:38.633136 22765 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:38.633618 22765 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:38.637889 22765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "e8e2c684993d46dbacb6996192ddd067"
format_stamp: "Formatted at 2025-09-05 13:57:38 on dist-test-slave-g001"
I20250905 13:57:38.638945 22765 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "e8e2c684993d46dbacb6996192ddd067"
format_stamp: "Formatted at 2025-09-05 13:57:38 on dist-test-slave-g001"
I20250905 13:57:38.645823 22765 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250905 13:57:38.651124 22782 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:38.652105 22765 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 13:57:38.652419 22765 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "e8e2c684993d46dbacb6996192ddd067"
format_stamp: "Formatted at 2025-09-05 13:57:38 on dist-test-slave-g001"
I20250905 13:57:38.652778 22765 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:38.719310 22765 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:38.720639 22765 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:38.721040 22765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:38.785949 22765 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:35679
I20250905 13:57:38.786013 22833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:35679 every 8 connection(s)
I20250905 13:57:38.788576 22765 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 13:57:38.793987 22834 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:38.796034 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 22765
I20250905 13:57:38.796609 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 13:57:38.814421 22834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067: Bootstrap starting.
I20250905 13:57:38.820010 22834 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:38.821604 22834 log.cc:826] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:38.826162 22834 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067: No bootstrap required, opened a new log
I20250905 13:57:38.842010 22834 raft_consensus.cc:357] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e2c684993d46dbacb6996192ddd067" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35679 } }
I20250905 13:57:38.842803 22834 raft_consensus.cc:383] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:38.843152 22834 raft_consensus.cc:738] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e2c684993d46dbacb6996192ddd067, State: Initialized, Role: FOLLOWER
I20250905 13:57:38.843775 22834 consensus_queue.cc:260] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e2c684993d46dbacb6996192ddd067" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35679 } }
I20250905 13:57:38.844201 22834 raft_consensus.cc:397] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:57:38.844391 22834 raft_consensus.cc:491] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:57:38.844615 22834 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:38.848201 22834 raft_consensus.cc:513] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e2c684993d46dbacb6996192ddd067" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35679 } }
I20250905 13:57:38.848951 22834 leader_election.cc:304] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e8e2c684993d46dbacb6996192ddd067; no voters:
I20250905 13:57:38.850665 22834 leader_election.cc:290] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:57:38.851380 22839 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:57:38.853567 22839 raft_consensus.cc:695] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [term 1 LEADER]: Becoming Leader. State: Replica: e8e2c684993d46dbacb6996192ddd067, State: Running, Role: LEADER
I20250905 13:57:38.854202 22839 consensus_queue.cc:237] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e2c684993d46dbacb6996192ddd067" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35679 } }
I20250905 13:57:38.854686 22834 sys_catalog.cc:564] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:57:38.864815 22840 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e8e2c684993d46dbacb6996192ddd067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e2c684993d46dbacb6996192ddd067" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35679 } } }
I20250905 13:57:38.866679 22840 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [sys.catalog]: This master's current role is: LEADER
I20250905 13:57:38.866036 22841 sys_catalog.cc:455] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e8e2c684993d46dbacb6996192ddd067. Latest consensus state: current_term: 1 leader_uuid: "e8e2c684993d46dbacb6996192ddd067" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e8e2c684993d46dbacb6996192ddd067" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35679 } } }
I20250905 13:57:38.868348 22841 sys_catalog.cc:458] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067 [sys.catalog]: This master's current role is: LEADER
I20250905 13:57:38.869881 22847 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:57:38.880299 22847 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:57:38.896806 22847 catalog_manager.cc:1349] Generated new cluster ID: 6dd83cdf6d5f468c9761f9eb233e90c4
I20250905 13:57:38.897094 22847 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:57:38.916168 22847 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 13:57:38.917402 22847 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:57:38.932564 22847 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e8e2c684993d46dbacb6996192ddd067: Generated new TSK 0
I20250905 13:57:38.933363 22847 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 13:57:38.947981 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:0
--local_ip_for_outbound_sockets=127.21.98.193
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35679
--builtin_ntp_servers=127.21.98.212:43347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
W20250905 13:57:39.245666 22858 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250905 13:57:39.246284 22858 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:39.246551 22858 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:39.247051 22858 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:39.277482 22858 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:39.278375 22858 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 13:57:39.311903 22858 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:39.313226 22858 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:39.314867 22858 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:39.327282 22864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:40.729856 22863 debug-util.cc:398] Leaking SignalData structure 0x7b08000068a0 after lost signal to thread 22858
W20250905 13:57:40.789479 22858 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.460s user 0.001s sys 0.000s
W20250905 13:57:39.328172 22865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:40.790503 22858 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.461s user 0.002s sys 0.001s
W20250905 13:57:40.792173 22867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:57:40.796828 22858 server_base.cc:1047] running on GCE node
I20250905 13:57:40.798023 22858 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:40.799908 22858 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:40.801213 22858 hybrid_clock.cc:648] HybridClock initialized: now 1757080660801169 us; error 39 us; skew 500 ppm
I20250905 13:57:40.801942 22858 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:40.807300 22858 webserver.cc:480] Webserver started at http://127.21.98.193:45561/ using document root <none> and password file <none>
I20250905 13:57:40.808148 22858 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:40.808387 22858 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:40.808795 22858 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:40.812896 22858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "4e6448981a1849df85a49eeeb68cb0f2"
format_stamp: "Formatted at 2025-09-05 13:57:40 on dist-test-slave-g001"
I20250905 13:57:40.813928 22858 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "4e6448981a1849df85a49eeeb68cb0f2"
format_stamp: "Formatted at 2025-09-05 13:57:40 on dist-test-slave-g001"
I20250905 13:57:40.820336 22858 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250905 13:57:40.825460 22874 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:40.826311 22858 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 13:57:40.826587 22858 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "4e6448981a1849df85a49eeeb68cb0f2"
format_stamp: "Formatted at 2025-09-05 13:57:40 on dist-test-slave-g001"
I20250905 13:57:40.826864 22858 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:40.876684 22858 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:40.877909 22858 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:40.878266 22858 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:40.880520 22858 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:57:40.884290 22858 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:57:40.884456 22858 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:40.884714 22858 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:57:40.884842 22858 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:41.010426 22858 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:33111
I20250905 13:57:41.010641 22986 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:33111 every 8 connection(s)
I20250905 13:57:41.013402 22858 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 13:57:41.022172 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 22858
I20250905 13:57:41.022539 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 13:57:41.027843 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.194:0
--local_ip_for_outbound_sockets=127.21.98.194
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35679
--builtin_ntp_servers=127.21.98.212:43347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250905 13:57:41.033120 22987 heartbeater.cc:344] Connected to a master server at 127.21.98.254:35679
I20250905 13:57:41.033509 22987 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:41.034348 22987 heartbeater.cc:507] Master 127.21.98.254:35679 requested a full tablet report, sending...
I20250905 13:57:41.036770 22799 ts_manager.cc:194] Registered new tserver with Master: 4e6448981a1849df85a49eeeb68cb0f2 (127.21.98.193:33111)
I20250905 13:57:41.038563 22799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:40245
W20250905 13:57:41.363857 22991 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250905 13:57:41.364434 22991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:41.364637 22991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:41.365052 22991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:41.393443 22991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:41.394127 22991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.194
I20250905 13:57:41.428036 22991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.194
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:41.429332 22991 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:41.430796 22991 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:41.441587 22997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:57:42.041601 22987 heartbeater.cc:499] Master 127.21.98.254:35679 was elected leader, sending a full tablet report...
W20250905 13:57:41.443053 22998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:42.569233 23000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:42.571444 22999 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1124 milliseconds
I20250905 13:57:42.571521 22991 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:42.572661 22991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:42.574707 22991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:42.576040 22991 hybrid_clock.cc:648] HybridClock initialized: now 1757080662576009 us; error 44 us; skew 500 ppm
I20250905 13:57:42.576835 22991 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:42.582281 22991 webserver.cc:480] Webserver started at http://127.21.98.194:42155/ using document root <none> and password file <none>
I20250905 13:57:42.583235 22991 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:42.583464 22991 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:42.583868 22991 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:42.587889 22991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "1291f4379ec642458981b557d906ffc4"
format_stamp: "Formatted at 2025-09-05 13:57:42 on dist-test-slave-g001"
I20250905 13:57:42.588903 22991 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "1291f4379ec642458981b557d906ffc4"
format_stamp: "Formatted at 2025-09-05 13:57:42 on dist-test-slave-g001"
I20250905 13:57:42.595722 22991 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20250905 13:57:42.601051 23007 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:42.601981 22991 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.001s
I20250905 13:57:42.602301 22991 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "1291f4379ec642458981b557d906ffc4"
format_stamp: "Formatted at 2025-09-05 13:57:42 on dist-test-slave-g001"
I20250905 13:57:42.602756 22991 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:42.651530 22991 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:42.652964 22991 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:42.653343 22991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:42.655637 22991 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:57:42.659734 22991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:57:42.659917 22991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:42.660120 22991 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:57:42.660235 22991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:42.789561 22991 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.194:33371
I20250905 13:57:42.789630 23119 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.194:33371 every 8 connection(s)
I20250905 13:57:42.793048 22991 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 13:57:42.794252 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 22991
I20250905 13:57:42.794631 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 13:57:42.802487 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.195:0
--local_ip_for_outbound_sockets=127.21.98.195
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35679
--builtin_ntp_servers=127.21.98.212:43347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false with env {}
I20250905 13:57:42.833305 23120 heartbeater.cc:344] Connected to a master server at 127.21.98.254:35679
I20250905 13:57:42.833745 23120 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:42.834851 23120 heartbeater.cc:507] Master 127.21.98.254:35679 requested a full tablet report, sending...
I20250905 13:57:42.837116 22799 ts_manager.cc:194] Registered new tserver with Master: 1291f4379ec642458981b557d906ffc4 (127.21.98.194:33371)
I20250905 13:57:42.838338 22799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.194:54339
W20250905 13:57:43.097738 23124 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20250905 13:57:43.098299 23124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:43.098522 23124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:43.098966 23124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:43.126390 23124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:43.127228 23124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.195
I20250905 13:57:43.159708 23124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.195
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:43.160856 23124 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:43.162310 23124 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:43.173066 23130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:57:43.841240 23120 heartbeater.cc:499] Master 127.21.98.254:35679 was elected leader, sending a full tablet report...
W20250905 13:57:43.173556 23131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:44.370710 23133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:44.373921 23124 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.201s user 0.354s sys 0.843s
W20250905 13:57:44.374271 23124 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.201s user 0.354s sys 0.843s
W20250905 13:57:44.374370 23132 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1195 milliseconds
I20250905 13:57:44.374550 23124 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:44.376235 23124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:44.379527 23124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:44.381026 23124 hybrid_clock.cc:648] HybridClock initialized: now 1757080664380989 us; error 28 us; skew 500 ppm
I20250905 13:57:44.382133 23124 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:44.390771 23124 webserver.cc:480] Webserver started at http://127.21.98.195:35895/ using document root <none> and password file <none>
I20250905 13:57:44.392086 23124 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:44.392409 23124 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:44.393033 23124 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:44.400271 23124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "ab408835d8364cdcb601be502b22e864"
format_stamp: "Formatted at 2025-09-05 13:57:44 on dist-test-slave-g001"
I20250905 13:57:44.401710 23124 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "ab408835d8364cdcb601be502b22e864"
format_stamp: "Formatted at 2025-09-05 13:57:44 on dist-test-slave-g001"
I20250905 13:57:44.411075 23124 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.010s sys 0.002s
I20250905 13:57:44.419456 23140 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:44.420513 23124 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250905 13:57:44.420899 23124 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "ab408835d8364cdcb601be502b22e864"
format_stamp: "Formatted at 2025-09-05 13:57:44 on dist-test-slave-g001"
I20250905 13:57:44.421308 23124 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:44.538292 23124 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:44.539781 23124 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:44.540140 23124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:44.542443 23124 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:57:44.547185 23124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:57:44.547394 23124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:44.547588 23124 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:57:44.547716 23124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:44.675056 23124 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.195:43025
I20250905 13:57:44.675153 23252 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.195:43025 every 8 connection(s)
I20250905 13:57:44.677613 23124 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 13:57:44.687697 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 23124
I20250905 13:57:44.688019 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestGracefulSpecificLeaderStepDown.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 13:57:44.700238 23253 heartbeater.cc:344] Connected to a master server at 127.21.98.254:35679
I20250905 13:57:44.700606 23253 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:44.701473 23253 heartbeater.cc:507] Master 127.21.98.254:35679 requested a full tablet report, sending...
I20250905 13:57:44.703365 22799 ts_manager.cc:194] Registered new tserver with Master: ab408835d8364cdcb601be502b22e864 (127.21.98.195:43025)
I20250905 13:57:44.704725 22799 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.195:58273
I20250905 13:57:44.706921 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:57:44.739557 22799 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39158:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250905 13:57:44.757369 22799 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 13:57:44.835985 23188 tablet_service.cc:1468] Processing CreateTablet for tablet 8d9c1d414ac04dc69d707f331d7bf0b6 (DEFAULT_TABLE table=TestTable [id=3bd73a8655c6417881c4f35066993714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:57:44.838701 23188 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9c1d414ac04dc69d707f331d7bf0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:44.846474 23055 tablet_service.cc:1468] Processing CreateTablet for tablet 8d9c1d414ac04dc69d707f331d7bf0b6 (DEFAULT_TABLE table=TestTable [id=3bd73a8655c6417881c4f35066993714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:57:44.842113 22922 tablet_service.cc:1468] Processing CreateTablet for tablet 8d9c1d414ac04dc69d707f331d7bf0b6 (DEFAULT_TABLE table=TestTable [id=3bd73a8655c6417881c4f35066993714]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:57:44.848317 22922 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9c1d414ac04dc69d707f331d7bf0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:44.850106 23055 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d9c1d414ac04dc69d707f331d7bf0b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:44.875645 23273 tablet_bootstrap.cc:492] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2: Bootstrap starting.
I20250905 13:57:44.876403 23272 tablet_bootstrap.cc:492] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864: Bootstrap starting.
I20250905 13:57:44.880643 23274 tablet_bootstrap.cc:492] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4: Bootstrap starting.
I20250905 13:57:44.882385 23272 tablet_bootstrap.cc:654] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:44.882434 23273 tablet_bootstrap.cc:654] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:44.884385 23273 log.cc:826] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:44.884702 23272 log.cc:826] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:44.888399 23274 tablet_bootstrap.cc:654] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:44.890673 23274 log.cc:826] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:44.891162 23273 tablet_bootstrap.cc:492] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2: No bootstrap required, opened a new log
I20250905 13:57:44.891501 23273 ts_tablet_manager.cc:1397] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2: Time spent bootstrapping tablet: real 0.016s user 0.012s sys 0.004s
I20250905 13:57:44.892699 23272 tablet_bootstrap.cc:492] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864: No bootstrap required, opened a new log
I20250905 13:57:44.893196 23272 ts_tablet_manager.cc:1397] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20250905 13:57:44.895767 23274 tablet_bootstrap.cc:492] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4: No bootstrap required, opened a new log
I20250905 13:57:44.896220 23274 ts_tablet_manager.cc:1397] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20250905 13:57:44.907908 23273 raft_consensus.cc:357] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:44.908505 23273 raft_consensus.cc:738] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6448981a1849df85a49eeeb68cb0f2, State: Initialized, Role: FOLLOWER
I20250905 13:57:44.909116 23273 consensus_queue.cc:260] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:44.914530 23273 ts_tablet_manager.cc:1428] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2: Time spent starting tablet: real 0.023s user 0.019s sys 0.002s
I20250905 13:57:44.917125 23272 raft_consensus.cc:357] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:44.917979 23272 raft_consensus.cc:738] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab408835d8364cdcb601be502b22e864, State: Initialized, Role: FOLLOWER
I20250905 13:57:44.918633 23272 consensus_queue.cc:260] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:44.921705 23253 heartbeater.cc:499] Master 127.21.98.254:35679 was elected leader, sending a full tablet report...
I20250905 13:57:44.920656 23274 raft_consensus.cc:357] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:44.922238 23274 raft_consensus.cc:738] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1291f4379ec642458981b557d906ffc4, State: Initialized, Role: FOLLOWER
I20250905 13:57:44.922645 23272 ts_tablet_manager.cc:1428] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864: Time spent starting tablet: real 0.029s user 0.024s sys 0.004s
I20250905 13:57:44.922996 23274 consensus_queue.cc:260] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:44.926890 23274 ts_tablet_manager.cc:1428] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4: Time spent starting tablet: real 0.030s user 0.018s sys 0.011s
W20250905 13:57:44.933823 23254 tablet.cc:2378] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:57:44.937371 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:57:44.940263 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 4e6448981a1849df85a49eeeb68cb0f2 to finish bootstrapping
I20250905 13:57:44.951385 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1291f4379ec642458981b557d906ffc4 to finish bootstrapping
I20250905 13:57:44.961066 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver ab408835d8364cdcb601be502b22e864 to finish bootstrapping
I20250905 13:57:44.998546 22942 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8d9c1d414ac04dc69d707f331d7bf0b6"
dest_uuid: "4e6448981a1849df85a49eeeb68cb0f2"
from {username='slave'} at 127.0.0.1:53534
I20250905 13:57:44.999076 22942 raft_consensus.cc:491] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 13:57:44.999348 22942 raft_consensus.cc:3058] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:45.003365 22942 raft_consensus.cc:513] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:45.005302 22942 leader_election.cc:290] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [CANDIDATE]: Term 1 election: Requested vote from peers ab408835d8364cdcb601be502b22e864 (127.21.98.195:43025), 1291f4379ec642458981b557d906ffc4 (127.21.98.194:33371)
I20250905 13:57:45.013904 21899 cluster_itest_util.cc:257] Not converged past 1 yet: 0.0 0.0 0.0
I20250905 13:57:45.017791 23075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d9c1d414ac04dc69d707f331d7bf0b6" candidate_uuid: "4e6448981a1849df85a49eeeb68cb0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1291f4379ec642458981b557d906ffc4"
I20250905 13:57:45.018350 23075 raft_consensus.cc:3058] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:45.018651 23208 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d9c1d414ac04dc69d707f331d7bf0b6" candidate_uuid: "4e6448981a1849df85a49eeeb68cb0f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "ab408835d8364cdcb601be502b22e864"
I20250905 13:57:45.019317 23208 raft_consensus.cc:3058] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [term 0 FOLLOWER]: Advancing to term 1
W20250905 13:57:45.024303 22988 tablet.cc:2378] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:57:45.025049 23075 raft_consensus.cc:2466] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e6448981a1849df85a49eeeb68cb0f2 in term 1.
I20250905 13:57:45.026006 23208 raft_consensus.cc:2466] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e6448981a1849df85a49eeeb68cb0f2 in term 1.
I20250905 13:57:45.026190 22876 leader_election.cc:304] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1291f4379ec642458981b557d906ffc4, 4e6448981a1849df85a49eeeb68cb0f2; no voters:
I20250905 13:57:45.027215 23278 raft_consensus.cc:2802] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:57:45.029121 23278 raft_consensus.cc:695] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 1 LEADER]: Becoming Leader. State: Replica: 4e6448981a1849df85a49eeeb68cb0f2, State: Running, Role: LEADER
I20250905 13:57:45.030040 23278 consensus_queue.cc:237] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:45.042479 22797 catalog_manager.cc:5582] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e6448981a1849df85a49eeeb68cb0f2 (127.21.98.193). New cstate: current_term: 1 leader_uuid: "4e6448981a1849df85a49eeeb68cb0f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } health_report { overall_health: UNKNOWN } } }
W20250905 13:57:45.057406 23121 tablet.cc:2378] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:57:45.119063 21899 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250905 13:57:45.323704 21899 cluster_itest_util.cc:257] Not converged past 1 yet: 1.1 0.0 0.0
I20250905 13:57:45.596923 23287 consensus_queue.cc:1035] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 13:57:45.614441 23293 consensus_queue.cc:1035] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 13:57:47.203171 22942 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8d9c1d414ac04dc69d707f331d7bf0b6"
dest_uuid: "4e6448981a1849df85a49eeeb68cb0f2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35864
I20250905 13:57:47.203686 22942 raft_consensus.cc:604] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 1 LEADER]: Received request to transfer leadership
I20250905 13:57:47.289386 23322 raft_consensus.cc:991] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2: : Instructing follower ab408835d8364cdcb601be502b22e864 to start an election
I20250905 13:57:47.289707 23311 raft_consensus.cc:1079] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 1 LEADER]: Signalling peer ab408835d8364cdcb601be502b22e864 to start an election
I20250905 13:57:47.290875 23208 tablet_service.cc:1940] Received Run Leader Election RPC: tablet_id: "8d9c1d414ac04dc69d707f331d7bf0b6"
dest_uuid: "ab408835d8364cdcb601be502b22e864"
from {username='slave'} at 127.21.98.193:51097
I20250905 13:57:47.291350 23208 raft_consensus.cc:491] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20250905 13:57:47.291574 23208 raft_consensus.cc:3058] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [term 1 FOLLOWER]: Advancing to term 2
I20250905 13:57:47.295383 23208 raft_consensus.cc:513] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:47.297554 23208 leader_election.cc:290] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [CANDIDATE]: Term 2 election: Requested vote from peers 4e6448981a1849df85a49eeeb68cb0f2 (127.21.98.193:33111), 1291f4379ec642458981b557d906ffc4 (127.21.98.194:33371)
I20250905 13:57:47.306943 22942 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d9c1d414ac04dc69d707f331d7bf0b6" candidate_uuid: "ab408835d8364cdcb601be502b22e864" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "4e6448981a1849df85a49eeeb68cb0f2"
I20250905 13:57:47.307519 22942 raft_consensus.cc:3053] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 13:57:47.307732 22942 raft_consensus.cc:738] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e6448981a1849df85a49eeeb68cb0f2, State: Running, Role: LEADER
I20250905 13:57:47.307952 23075 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d9c1d414ac04dc69d707f331d7bf0b6" candidate_uuid: "ab408835d8364cdcb601be502b22e864" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: true dest_uuid: "1291f4379ec642458981b557d906ffc4"
I20250905 13:57:47.308349 23075 raft_consensus.cc:3058] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4 [term 1 FOLLOWER]: Advancing to term 2
I20250905 13:57:47.308157 22942 consensus_queue.cc:260] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:47.309019 22942 raft_consensus.cc:3058] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 1 FOLLOWER]: Advancing to term 2
I20250905 13:57:47.311902 23075 raft_consensus.cc:2466] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab408835d8364cdcb601be502b22e864 in term 2.
I20250905 13:57:47.312868 22942 raft_consensus.cc:2466] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab408835d8364cdcb601be502b22e864 in term 2.
I20250905 13:57:47.312806 23142 leader_election.cc:304] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1291f4379ec642458981b557d906ffc4, ab408835d8364cdcb601be502b22e864; no voters:
I20250905 13:57:47.314687 23326 raft_consensus.cc:2802] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 13:57:47.315882 23326 raft_consensus.cc:695] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [term 2 LEADER]: Becoming Leader. State: Replica: ab408835d8364cdcb601be502b22e864, State: Running, Role: LEADER
I20250905 13:57:47.316576 23326 consensus_queue.cc:237] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } }
I20250905 13:57:47.323488 22797 catalog_manager.cc:5582] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 reported cstate change: term changed from 1 to 2, leader changed from 4e6448981a1849df85a49eeeb68cb0f2 (127.21.98.193) to ab408835d8364cdcb601be502b22e864 (127.21.98.195). New cstate: current_term: 2 leader_uuid: "ab408835d8364cdcb601be502b22e864" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab408835d8364cdcb601be502b22e864" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 } health_report { overall_health: UNKNOWN } } }
I20250905 13:57:47.866684 23075 raft_consensus.cc:1273] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 1291f4379ec642458981b557d906ffc4 [term 2 FOLLOWER]: Refusing update from remote peer ab408835d8364cdcb601be502b22e864: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 13:57:47.867759 23335 consensus_queue.cc:1035] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1291f4379ec642458981b557d906ffc4" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 13:57:47.879537 22942 raft_consensus.cc:1273] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 2 FOLLOWER]: Refusing update from remote peer ab408835d8364cdcb601be502b22e864: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 13:57:47.880916 23340 consensus_queue.cc:1035] T 8d9c1d414ac04dc69d707f331d7bf0b6 P ab408835d8364cdcb601be502b22e864 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6448981a1849df85a49eeeb68cb0f2" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 33111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 13:57:49.818600 22942 tablet_service.cc:1968] Received LeaderStepDown RPC: tablet_id: "8d9c1d414ac04dc69d707f331d7bf0b6"
dest_uuid: "4e6448981a1849df85a49eeeb68cb0f2"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35876
I20250905 13:57:49.819296 22942 raft_consensus.cc:604] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 2 FOLLOWER]: Received request to transfer leadership
I20250905 13:57:49.819664 22942 raft_consensus.cc:612] T 8d9c1d414ac04dc69d707f331d7bf0b6 P 4e6448981a1849df85a49eeeb68cb0f2 [term 2 FOLLOWER]: Rejecting request to transer leadership while not leader
I20250905 13:57:50.904886 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 22858
I20250905 13:57:50.927875 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 22991
I20250905 13:57:50.951534 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 23124
I20250905 13:57:50.977519 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 22765
2025-09-05T13:57:50Z chronyd exiting
[ OK ] AdminCliTest.TestGracefulSpecificLeaderStepDown (13908 ms)
[ RUN ] AdminCliTest.TestDescribeTableColumnFlags
I20250905 13:57:51.028554 21899 test_util.cc:276] Using random seed: 960300338
I20250905 13:57:51.032042 21899 ts_itest-base.cc:115] Starting cluster with:
I20250905 13:57:51.032173 21899 ts_itest-base.cc:116] --------------
I20250905 13:57:51.032320 21899 ts_itest-base.cc:117] 3 tablet servers
I20250905 13:57:51.032434 21899 ts_itest-base.cc:118] 3 replicas per TS
I20250905 13:57:51.032548 21899 ts_itest-base.cc:119] --------------
2025-09-05T13:57:51Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T13:57:51Z Disabled control of system clock
I20250905 13:57:51.065310 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:34619
--webserver_interface=127.21.98.254
--webserver_port=0
--builtin_ntp_servers=127.21.98.212:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:34619 with env {}
W20250905 13:57:51.345324 23374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:51.345796 23374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:51.346140 23374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:51.373474 23374 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:57:51.373744 23374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:51.373967 23374 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:57:51.374145 23374 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:57:51.406999 23374 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:34619
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:34619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:51.408205 23374 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:51.409714 23374 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:51.419674 23380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:51.420387 23381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:52.501281 23382 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250905 13:57:52.502593 23374 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250905 13:57:52.502609 23383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:57:52.505148 23374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:52.507386 23374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:52.508783 23374 hybrid_clock.cc:648] HybridClock initialized: now 1757080672508732 us; error 34 us; skew 500 ppm
I20250905 13:57:52.509639 23374 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:52.515216 23374 webserver.cc:480] Webserver started at http://127.21.98.254:33911/ using document root <none> and password file <none>
I20250905 13:57:52.516094 23374 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:52.516294 23374 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:52.516675 23374 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:52.522032 23374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "36974a9b003b43a3b3d8724454bab499"
format_stamp: "Formatted at 2025-09-05 13:57:52 on dist-test-slave-g001"
I20250905 13:57:52.523023 23374 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "36974a9b003b43a3b3d8724454bab499"
format_stamp: "Formatted at 2025-09-05 13:57:52 on dist-test-slave-g001"
I20250905 13:57:52.529163 23374 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250905 13:57:52.534029 23390 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:52.535007 23374 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 13:57:52.535362 23374 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "36974a9b003b43a3b3d8724454bab499"
format_stamp: "Formatted at 2025-09-05 13:57:52 on dist-test-slave-g001"
I20250905 13:57:52.535673 23374 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:52.586241 23374 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:52.587638 23374 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:52.588069 23374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:52.654404 23374 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:34619
I20250905 13:57:52.654449 23441 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:34619 every 8 connection(s)
I20250905 13:57:52.657456 23374 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 13:57:52.661618 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 23374
I20250905 13:57:52.662065 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 13:57:52.663975 23442 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:52.687721 23442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499: Bootstrap starting.
I20250905 13:57:52.692979 23442 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:52.694628 23442 log.cc:826] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:52.699036 23442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499: No bootstrap required, opened a new log
I20250905 13:57:52.714345 23442 raft_consensus.cc:357] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36974a9b003b43a3b3d8724454bab499" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 34619 } }
I20250905 13:57:52.714879 23442 raft_consensus.cc:383] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:52.715134 23442 raft_consensus.cc:738] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36974a9b003b43a3b3d8724454bab499, State: Initialized, Role: FOLLOWER
I20250905 13:57:52.715900 23442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36974a9b003b43a3b3d8724454bab499" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 34619 } }
I20250905 13:57:52.716432 23442 raft_consensus.cc:397] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:57:52.716684 23442 raft_consensus.cc:491] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:57:52.716953 23442 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:52.720321 23442 raft_consensus.cc:513] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36974a9b003b43a3b3d8724454bab499" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 34619 } }
I20250905 13:57:52.720858 23442 leader_election.cc:304] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36974a9b003b43a3b3d8724454bab499; no voters:
I20250905 13:57:52.722278 23442 leader_election.cc:290] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:57:52.722846 23447 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:57:52.725510 23447 raft_consensus.cc:695] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [term 1 LEADER]: Becoming Leader. State: Replica: 36974a9b003b43a3b3d8724454bab499, State: Running, Role: LEADER
I20250905 13:57:52.726575 23442 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:57:52.726313 23447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36974a9b003b43a3b3d8724454bab499" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 34619 } }
I20250905 13:57:52.735793 23449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36974a9b003b43a3b3d8724454bab499. Latest consensus state: current_term: 1 leader_uuid: "36974a9b003b43a3b3d8724454bab499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36974a9b003b43a3b3d8724454bab499" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 34619 } } }
I20250905 13:57:52.736511 23449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [sys.catalog]: This master's current role is: LEADER
I20250905 13:57:52.736701 23448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36974a9b003b43a3b3d8724454bab499" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36974a9b003b43a3b3d8724454bab499" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 34619 } } }
I20250905 13:57:52.737375 23448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499 [sys.catalog]: This master's current role is: LEADER
I20250905 13:57:52.748255 23454 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:57:52.758014 23454 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:57:52.772469 23454 catalog_manager.cc:1349] Generated new cluster ID: e796aea08bab4c5fa9a780c4c7402ea2
I20250905 13:57:52.772729 23454 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:57:52.790567 23454 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 13:57:52.792307 23454 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:57:52.807466 23454 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 36974a9b003b43a3b3d8724454bab499: Generated new TSK 0
I20250905 13:57:52.808372 23454 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 13:57:52.818581 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:0
--local_ip_for_outbound_sockets=127.21.98.193
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:34619
--builtin_ntp_servers=127.21.98.212:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250905 13:57:53.103446 23466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:53.103958 23466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:53.104465 23466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:53.132894 23466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:53.133713 23466 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 13:57:53.164263 23466 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:34619
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:53.165707 23466 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:53.167511 23466 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:53.180543 23472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:53.183013 23473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:54.500490 23474 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1316 milliseconds
W20250905 13:57:54.501381 23475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:54.505635 23466 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.324s user 0.458s sys 0.862s
W20250905 13:57:54.505947 23466 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.324s user 0.458s sys 0.862s
I20250905 13:57:54.506234 23466 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:54.507566 23466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:54.510130 23466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:54.511628 23466 hybrid_clock.cc:648] HybridClock initialized: now 1757080674511542 us; error 79 us; skew 500 ppm
I20250905 13:57:54.512665 23466 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:54.521198 23466 webserver.cc:480] Webserver started at http://127.21.98.193:44529/ using document root <none> and password file <none>
I20250905 13:57:54.522415 23466 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:54.522679 23466 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:54.523301 23466 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:54.529875 23466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5aeb3545a65448fea2bf75704fd8a1ab"
format_stamp: "Formatted at 2025-09-05 13:57:54 on dist-test-slave-g001"
I20250905 13:57:54.531276 23466 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5aeb3545a65448fea2bf75704fd8a1ab"
format_stamp: "Formatted at 2025-09-05 13:57:54 on dist-test-slave-g001"
I20250905 13:57:54.540639 23466 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.004s sys 0.004s
I20250905 13:57:54.548254 23482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:54.549392 23466 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20250905 13:57:54.549752 23466 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5aeb3545a65448fea2bf75704fd8a1ab"
format_stamp: "Formatted at 2025-09-05 13:57:54 on dist-test-slave-g001"
I20250905 13:57:54.550150 23466 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:54.654671 23466 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:54.656486 23466 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:54.656948 23466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:54.659847 23466 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:57:54.664950 23466 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:57:54.665171 23466 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:54.665450 23466 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:57:54.665663 23466 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:54.827154 23466 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:38791
I20250905 13:57:54.827241 23594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:38791 every 8 connection(s)
I20250905 13:57:54.829891 23466 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 13:57:54.837527 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 23466
I20250905 13:57:54.837947 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 13:57:54.847553 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.194:0
--local_ip_for_outbound_sockets=127.21.98.194
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:34619
--builtin_ntp_servers=127.21.98.212:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:57:54.873927 23595 heartbeater.cc:344] Connected to a master server at 127.21.98.254:34619
I20250905 13:57:54.874459 23595 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:54.875751 23595 heartbeater.cc:507] Master 127.21.98.254:34619 requested a full tablet report, sending...
I20250905 13:57:54.878753 23407 ts_manager.cc:194] Registered new tserver with Master: 5aeb3545a65448fea2bf75704fd8a1ab (127.21.98.193:38791)
I20250905 13:57:54.882138 23407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:58707
W20250905 13:57:55.140980 23599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:55.141425 23599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:55.141867 23599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:55.169116 23599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:55.169883 23599 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.194
I20250905 13:57:55.200474 23599 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:34619
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.194
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:55.201752 23599 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:55.203336 23599 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:55.213008 23605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:57:55.886283 23595 heartbeater.cc:499] Master 127.21.98.254:34619 was elected leader, sending a full tablet report...
W20250905 13:57:55.214748 23606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:55.217711 23608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:56.270586 23607 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250905 13:57:56.270605 23599 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:56.273979 23599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:56.276074 23599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:56.277447 23599 hybrid_clock.cc:648] HybridClock initialized: now 1757080676277397 us; error 50 us; skew 500 ppm
I20250905 13:57:56.278129 23599 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:56.283324 23599 webserver.cc:480] Webserver started at http://127.21.98.194:43379/ using document root <none> and password file <none>
I20250905 13:57:56.284204 23599 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:56.284420 23599 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:56.284821 23599 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:56.289014 23599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "46747e39d2fe422b983be3a6c560eee3"
format_stamp: "Formatted at 2025-09-05 13:57:56 on dist-test-slave-g001"
I20250905 13:57:56.289983 23599 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "46747e39d2fe422b983be3a6c560eee3"
format_stamp: "Formatted at 2025-09-05 13:57:56 on dist-test-slave-g001"
I20250905 13:57:56.296428 23599 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.000s sys 0.008s
I20250905 13:57:56.301115 23615 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:56.301980 23599 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 13:57:56.302270 23599 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "46747e39d2fe422b983be3a6c560eee3"
format_stamp: "Formatted at 2025-09-05 13:57:56 on dist-test-slave-g001"
I20250905 13:57:56.302567 23599 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:56.358117 23599 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:56.359364 23599 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:56.359728 23599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:56.361944 23599 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:57:56.365622 23599 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:57:56.365803 23599 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:56.366009 23599 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:57:56.366144 23599 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:56.490324 23599 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.194:41113
I20250905 13:57:56.490428 23727 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.194:41113 every 8 connection(s)
I20250905 13:57:56.492774 23599 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 13:57:56.497038 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 23599
I20250905 13:57:56.497539 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 13:57:56.503206 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.195:0
--local_ip_for_outbound_sockets=127.21.98.195
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:34619
--builtin_ntp_servers=127.21.98.212:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:57:56.512661 23728 heartbeater.cc:344] Connected to a master server at 127.21.98.254:34619
I20250905 13:57:56.513023 23728 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:56.513986 23728 heartbeater.cc:507] Master 127.21.98.254:34619 requested a full tablet report, sending...
I20250905 13:57:56.516108 23407 ts_manager.cc:194] Registered new tserver with Master: 46747e39d2fe422b983be3a6c560eee3 (127.21.98.194:41113)
I20250905 13:57:56.517833 23407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.194:57229
W20250905 13:57:56.784490 23732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:56.784906 23732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:57:56.785305 23732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:56.812275 23732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:57:56.812990 23732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.195
I20250905 13:57:56.843132 23732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:34619
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.195
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:57:56.844295 23732 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:57:56.845736 23732 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:57:56.856770 23738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:57:57.521078 23728 heartbeater.cc:499] Master 127.21.98.254:34619 was elected leader, sending a full tablet report...
W20250905 13:57:56.857635 23739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:57.964218 23740 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1104 milliseconds
W20250905 13:57:57.965119 23741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:57:57.967247 23732 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.110s user 0.390s sys 0.720s
W20250905 13:57:57.967509 23732 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.110s user 0.390s sys 0.720s
I20250905 13:57:57.967782 23732 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:57:57.968693 23732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:57:57.970743 23732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:57:57.972128 23732 hybrid_clock.cc:648] HybridClock initialized: now 1757080677972095 us; error 34 us; skew 500 ppm
I20250905 13:57:57.972811 23732 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:57:57.979633 23732 webserver.cc:480] Webserver started at http://127.21.98.195:43651/ using document root <none> and password file <none>
I20250905 13:57:57.980448 23732 fs_manager.cc:362] Metadata directory not provided
I20250905 13:57:57.980641 23732 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:57:57.981105 23732 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:57:57.985179 23732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d"
format_stamp: "Formatted at 2025-09-05 13:57:57 on dist-test-slave-g001"
I20250905 13:57:57.986125 23732 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d"
format_stamp: "Formatted at 2025-09-05 13:57:57 on dist-test-slave-g001"
I20250905 13:57:57.992424 23732 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250905 13:57:57.998271 23749 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:57.999367 23732 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250905 13:57:57.999660 23732 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d"
format_stamp: "Formatted at 2025-09-05 13:57:57 on dist-test-slave-g001"
I20250905 13:57:57.999938 23732 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:57:58.072681 23732 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:57:58.073992 23732 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:57:58.074425 23732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:57:58.076709 23732 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:57:58.081133 23732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:57:58.081301 23732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:58.081517 23732 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:57:58.081671 23732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:57:58.203395 23732 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.195:43361
I20250905 13:57:58.203482 23861 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.195:43361 every 8 connection(s)
I20250905 13:57:58.205801 23732 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 13:57:58.215016 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 23732
I20250905 13:57:58.215616 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestDescribeTableColumnFlags.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 13:57:58.225978 23862 heartbeater.cc:344] Connected to a master server at 127.21.98.254:34619
I20250905 13:57:58.226351 23862 heartbeater.cc:461] Registering TS with master...
I20250905 13:57:58.227283 23862 heartbeater.cc:507] Master 127.21.98.254:34619 requested a full tablet report, sending...
I20250905 13:57:58.229230 23407 ts_manager.cc:194] Registered new tserver with Master: 3b0d1a154ac44337a0e2cf8a0f2f8e0d (127.21.98.195:43361)
I20250905 13:57:58.230381 23407 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.195:39773
I20250905 13:57:58.234802 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:57:58.264556 23407 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60486:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250905 13:57:58.281119 23407 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 13:57:58.331116 23663 tablet_service.cc:1468] Processing CreateTablet for tablet cddd86a956114ec6ad2a55ca1410035b (DEFAULT_TABLE table=TestTable [id=1c55753e5541410086e9d0c30f585e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:57:58.333074 23663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cddd86a956114ec6ad2a55ca1410035b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:58.336665 23797 tablet_service.cc:1468] Processing CreateTablet for tablet cddd86a956114ec6ad2a55ca1410035b (DEFAULT_TABLE table=TestTable [id=1c55753e5541410086e9d0c30f585e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:57:58.337800 23530 tablet_service.cc:1468] Processing CreateTablet for tablet cddd86a956114ec6ad2a55ca1410035b (DEFAULT_TABLE table=TestTable [id=1c55753e5541410086e9d0c30f585e90]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:57:58.339143 23797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cddd86a956114ec6ad2a55ca1410035b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:58.339478 23530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cddd86a956114ec6ad2a55ca1410035b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:58.357229 23881 tablet_bootstrap.cc:492] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: Bootstrap starting.
I20250905 13:57:58.362141 23882 tablet_bootstrap.cc:492] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3: Bootstrap starting.
I20250905 13:57:58.364302 23881 tablet_bootstrap.cc:654] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:58.365614 23883 tablet_bootstrap.cc:492] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab: Bootstrap starting.
I20250905 13:57:58.367461 23881 log.cc:826] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:58.369258 23882 tablet_bootstrap.cc:654] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:58.370908 23883 tablet_bootstrap.cc:654] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:58.371409 23882 log.cc:826] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:58.372455 23883 log.cc:826] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab: Log is configured to *not* fsync() on all Append() calls
I20250905 13:57:58.372850 23881 tablet_bootstrap.cc:492] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: No bootstrap required, opened a new log
I20250905 13:57:58.373504 23881 ts_tablet_manager.cc:1397] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.007s
I20250905 13:57:58.378211 23882 tablet_bootstrap.cc:492] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3: No bootstrap required, opened a new log
I20250905 13:57:58.378394 23883 tablet_bootstrap.cc:492] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab: No bootstrap required, opened a new log
I20250905 13:57:58.378549 23882 ts_tablet_manager.cc:1397] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20250905 13:57:58.378829 23883 ts_tablet_manager.cc:1397] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20250905 13:57:58.394369 23882 raft_consensus.cc:357] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.394955 23882 raft_consensus.cc:383] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:58.395226 23882 raft_consensus.cc:738] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46747e39d2fe422b983be3a6c560eee3, State: Initialized, Role: FOLLOWER
I20250905 13:57:58.395936 23882 consensus_queue.cc:260] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.398176 23881 raft_consensus.cc:357] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.399008 23881 raft_consensus.cc:383] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:58.399313 23881 raft_consensus.cc:738] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b0d1a154ac44337a0e2cf8a0f2f8e0d, State: Initialized, Role: FOLLOWER
I20250905 13:57:58.399677 23882 ts_tablet_manager.cc:1428] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3: Time spent starting tablet: real 0.021s user 0.021s sys 0.001s
I20250905 13:57:58.400204 23881 consensus_queue.cc:260] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.403017 23862 heartbeater.cc:499] Master 127.21.98.254:34619 was elected leader, sending a full tablet report...
I20250905 13:57:58.402758 23883 raft_consensus.cc:357] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.403635 23883 raft_consensus.cc:383] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:58.403932 23883 raft_consensus.cc:738] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aeb3545a65448fea2bf75704fd8a1ab, State: Initialized, Role: FOLLOWER
I20250905 13:57:58.404455 23881 ts_tablet_manager.cc:1428] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: Time spent starting tablet: real 0.031s user 0.027s sys 0.002s
I20250905 13:57:58.404851 23883 consensus_queue.cc:260] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.409050 23883 ts_tablet_manager.cc:1428] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab: Time spent starting tablet: real 0.030s user 0.028s sys 0.000s
W20250905 13:57:58.460301 23863 tablet.cc:2378] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 13:57:58.498971 23729 tablet.cc:2378] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:57:58.528591 23888 raft_consensus.cc:491] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:57:58.528945 23888 raft_consensus.cc:513] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.530721 23887 raft_consensus.cc:491] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:57:58.531193 23888 leader_election.cc:290] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46747e39d2fe422b983be3a6c560eee3 (127.21.98.194:41113), 5aeb3545a65448fea2bf75704fd8a1ab (127.21.98.193:38791)
I20250905 13:57:58.531188 23887 raft_consensus.cc:513] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.537968 23887 leader_election.cc:290] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aeb3545a65448fea2bf75704fd8a1ab (127.21.98.193:38791), 3b0d1a154ac44337a0e2cf8a0f2f8e0d (127.21.98.195:43361)
I20250905 13:57:58.542512 23683 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cddd86a956114ec6ad2a55ca1410035b" candidate_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46747e39d2fe422b983be3a6c560eee3" is_pre_election: true
I20250905 13:57:58.543262 23683 raft_consensus.cc:2466] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b0d1a154ac44337a0e2cf8a0f2f8e0d in term 0.
I20250905 13:57:58.544798 23753 leader_election.cc:304] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b0d1a154ac44337a0e2cf8a0f2f8e0d, 46747e39d2fe422b983be3a6c560eee3; no voters:
I20250905 13:57:58.545846 23888 raft_consensus.cc:2802] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 13:57:58.546236 23888 raft_consensus.cc:491] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 13:57:58.546563 23888 raft_consensus.cc:3058] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:58.554153 23550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cddd86a956114ec6ad2a55ca1410035b" candidate_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" is_pre_election: true
I20250905 13:57:58.554117 23888 raft_consensus.cc:513] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.554986 23550 raft_consensus.cc:2466] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b0d1a154ac44337a0e2cf8a0f2f8e0d in term 0.
I20250905 13:57:58.556751 23683 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cddd86a956114ec6ad2a55ca1410035b" candidate_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46747e39d2fe422b983be3a6c560eee3"
I20250905 13:57:58.557211 23683 raft_consensus.cc:3058] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:58.557715 23550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cddd86a956114ec6ad2a55ca1410035b" candidate_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aeb3545a65448fea2bf75704fd8a1ab"
I20250905 13:57:58.558168 23550 raft_consensus.cc:3058] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:58.560251 23888 leader_election.cc:290] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [CANDIDATE]: Term 1 election: Requested vote from peers 46747e39d2fe422b983be3a6c560eee3 (127.21.98.194:41113), 5aeb3545a65448fea2bf75704fd8a1ab (127.21.98.193:38791)
I20250905 13:57:58.560873 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cddd86a956114ec6ad2a55ca1410035b" candidate_uuid: "46747e39d2fe422b983be3a6c560eee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" is_pre_election: true
I20250905 13:57:58.561744 23817 raft_consensus.cc:2391] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 46747e39d2fe422b983be3a6c560eee3 in current term 1: Already voted for candidate 3b0d1a154ac44337a0e2cf8a0f2f8e0d in this term.
I20250905 13:57:58.563638 23549 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cddd86a956114ec6ad2a55ca1410035b" candidate_uuid: "46747e39d2fe422b983be3a6c560eee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" is_pre_election: true
I20250905 13:57:58.564265 23683 raft_consensus.cc:2466] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b0d1a154ac44337a0e2cf8a0f2f8e0d in term 1.
I20250905 13:57:58.564731 23550 raft_consensus.cc:2466] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b0d1a154ac44337a0e2cf8a0f2f8e0d in term 1.
I20250905 13:57:58.565073 23753 leader_election.cc:304] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b0d1a154ac44337a0e2cf8a0f2f8e0d, 46747e39d2fe422b983be3a6c560eee3; no voters:
I20250905 13:57:58.565132 23619 leader_election.cc:304] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 46747e39d2fe422b983be3a6c560eee3; no voters: 3b0d1a154ac44337a0e2cf8a0f2f8e0d, 5aeb3545a65448fea2bf75704fd8a1ab
I20250905 13:57:58.565606 23888 raft_consensus.cc:2802] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:57:58.565871 23887 raft_consensus.cc:2747] T cddd86a956114ec6ad2a55ca1410035b P 46747e39d2fe422b983be3a6c560eee3 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 13:57:58.567075 23888 raft_consensus.cc:695] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 1 LEADER]: Becoming Leader. State: Replica: 3b0d1a154ac44337a0e2cf8a0f2f8e0d, State: Running, Role: LEADER
I20250905 13:57:58.567763 23888 consensus_queue.cc:237] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.575966 23405 catalog_manager.cc:5582] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b0d1a154ac44337a0e2cf8a0f2f8e0d (127.21.98.195). New cstate: current_term: 1 leader_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } health_report { overall_health: HEALTHY } } }
W20250905 13:57:58.588922 23596 tablet.cc:2378] T cddd86a956114ec6ad2a55ca1410035b P 5aeb3545a65448fea2bf75704fd8a1ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:57:58.617779 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:57:58.620744 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 5aeb3545a65448fea2bf75704fd8a1ab to finish bootstrapping
I20250905 13:57:58.631359 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 46747e39d2fe422b983be3a6c560eee3 to finish bootstrapping
I20250905 13:57:58.641163 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 3b0d1a154ac44337a0e2cf8a0f2f8e0d to finish bootstrapping
I20250905 13:57:58.653118 23405 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60486:
name: "TestAnotherTable"
schema {
columns {
name: "foo"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "bar"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
comment: "comment for bar"
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "foo"
}
}
}
W20250905 13:57:58.655071 23405 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestAnotherTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 13:57:58.671679 23530 tablet_service.cc:1468] Processing CreateTablet for tablet 69a6b46d98224a82992626a208f7642a (DEFAULT_TABLE table=TestAnotherTable [id=cef9289a2c104c01b5059c093833063f]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250905 13:57:58.671994 23663 tablet_service.cc:1468] Processing CreateTablet for tablet 69a6b46d98224a82992626a208f7642a (DEFAULT_TABLE table=TestAnotherTable [id=cef9289a2c104c01b5059c093833063f]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250905 13:57:58.672511 23530 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a6b46d98224a82992626a208f7642a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:58.672991 23663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a6b46d98224a82992626a208f7642a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:58.680160 23797 tablet_service.cc:1468] Processing CreateTablet for tablet 69a6b46d98224a82992626a208f7642a (DEFAULT_TABLE table=TestAnotherTable [id=cef9289a2c104c01b5059c093833063f]), partition=RANGE (foo) PARTITION UNBOUNDED
I20250905 13:57:58.681118 23797 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69a6b46d98224a82992626a208f7642a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:57:58.682723 23883 tablet_bootstrap.cc:492] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab: Bootstrap starting.
I20250905 13:57:58.685222 23882 tablet_bootstrap.cc:492] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3: Bootstrap starting.
I20250905 13:57:58.687786 23883 tablet_bootstrap.cc:654] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:58.690476 23882 tablet_bootstrap.cc:654] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:58.694169 23881 tablet_bootstrap.cc:492] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: Bootstrap starting.
I20250905 13:57:58.699373 23881 tablet_bootstrap.cc:654] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: Neither blocks nor log segments found. Creating new log.
I20250905 13:57:58.702649 23883 tablet_bootstrap.cc:492] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab: No bootstrap required, opened a new log
I20250905 13:57:58.703182 23883 ts_tablet_manager.cc:1397] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.011s
I20250905 13:57:58.709367 23883 raft_consensus.cc:357] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.710211 23883 raft_consensus.cc:383] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:58.710542 23883 raft_consensus.cc:738] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5aeb3545a65448fea2bf75704fd8a1ab, State: Initialized, Role: FOLLOWER
I20250905 13:57:58.711463 23881 tablet_bootstrap.cc:492] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: No bootstrap required, opened a new log
I20250905 13:57:58.711827 23881 ts_tablet_manager.cc:1397] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.002s
I20250905 13:57:58.714031 23881 raft_consensus.cc:357] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.714664 23881 raft_consensus.cc:383] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:58.714915 23881 raft_consensus.cc:738] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b0d1a154ac44337a0e2cf8a0f2f8e0d, State: Initialized, Role: FOLLOWER
I20250905 13:57:58.715027 23883 consensus_queue.cc:260] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.715737 23881 consensus_queue.cc:260] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.716509 23882 tablet_bootstrap.cc:492] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3: No bootstrap required, opened a new log
I20250905 13:57:58.716912 23882 ts_tablet_manager.cc:1397] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3: Time spent bootstrapping tablet: real 0.032s user 0.017s sys 0.014s
I20250905 13:57:58.717159 23883 ts_tablet_manager.cc:1428] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20250905 13:57:58.719141 23881 ts_tablet_manager.cc:1428] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20250905 13:57:58.719300 23882 raft_consensus.cc:357] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.719894 23882 raft_consensus.cc:383] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:57:58.720213 23882 raft_consensus.cc:738] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46747e39d2fe422b983be3a6c560eee3, State: Initialized, Role: FOLLOWER
I20250905 13:57:58.720774 23882 consensus_queue.cc:260] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.722661 23882 ts_tablet_manager.cc:1428] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250905 13:57:58.761950 23887 raft_consensus.cc:491] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:57:58.762248 23887 raft_consensus.cc:513] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.763463 23887 leader_election.cc:290] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5aeb3545a65448fea2bf75704fd8a1ab (127.21.98.193:38791), 3b0d1a154ac44337a0e2cf8a0f2f8e0d (127.21.98.195:43361)
I20250905 13:57:58.764247 23550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69a6b46d98224a82992626a208f7642a" candidate_uuid: "46747e39d2fe422b983be3a6c560eee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" is_pre_election: true
I20250905 13:57:58.764422 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69a6b46d98224a82992626a208f7642a" candidate_uuid: "46747e39d2fe422b983be3a6c560eee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" is_pre_election: true
I20250905 13:57:58.764838 23550 raft_consensus.cc:2466] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46747e39d2fe422b983be3a6c560eee3 in term 0.
I20250905 13:57:58.764968 23817 raft_consensus.cc:2466] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 46747e39d2fe422b983be3a6c560eee3 in term 0.
I20250905 13:57:58.765736 23619 leader_election.cc:304] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46747e39d2fe422b983be3a6c560eee3, 5aeb3545a65448fea2bf75704fd8a1ab; no voters:
I20250905 13:57:58.766283 23887 raft_consensus.cc:2802] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 13:57:58.766508 23887 raft_consensus.cc:491] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 13:57:58.766714 23887 raft_consensus.cc:3058] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:58.770805 23887 raft_consensus.cc:513] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.771988 23887 leader_election.cc:290] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [CANDIDATE]: Term 1 election: Requested vote from peers 5aeb3545a65448fea2bf75704fd8a1ab (127.21.98.193:38791), 3b0d1a154ac44337a0e2cf8a0f2f8e0d (127.21.98.195:43361)
I20250905 13:57:58.772567 23550 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69a6b46d98224a82992626a208f7642a" candidate_uuid: "46747e39d2fe422b983be3a6c560eee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5aeb3545a65448fea2bf75704fd8a1ab"
I20250905 13:57:58.772729 23817 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "69a6b46d98224a82992626a208f7642a" candidate_uuid: "46747e39d2fe422b983be3a6c560eee3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d"
I20250905 13:57:58.772883 23550 raft_consensus.cc:3058] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:58.773103 23817 raft_consensus.cc:3058] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:57:58.776649 23550 raft_consensus.cc:2466] T 69a6b46d98224a82992626a208f7642a P 5aeb3545a65448fea2bf75704fd8a1ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46747e39d2fe422b983be3a6c560eee3 in term 1.
I20250905 13:57:58.777354 23817 raft_consensus.cc:2466] T 69a6b46d98224a82992626a208f7642a P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 46747e39d2fe422b983be3a6c560eee3 in term 1.
I20250905 13:57:58.777321 23619 leader_election.cc:304] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46747e39d2fe422b983be3a6c560eee3, 5aeb3545a65448fea2bf75704fd8a1ab; no voters:
I20250905 13:57:58.777773 23887 raft_consensus.cc:2802] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:57:58.779247 23887 raft_consensus.cc:695] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [term 1 LEADER]: Becoming Leader. State: Replica: 46747e39d2fe422b983be3a6c560eee3, State: Running, Role: LEADER
I20250905 13:57:58.780004 23887 consensus_queue.cc:237] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } } peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } }
I20250905 13:57:58.789352 23407 catalog_manager.cc:5582] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 46747e39d2fe422b983be3a6c560eee3 (127.21.98.194). New cstate: current_term: 1 leader_uuid: "46747e39d2fe422b983be3a6c560eee3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 } health_report { overall_health: UNKNOWN } } }
I20250905 13:57:59.027007 23888 consensus_queue.cc:1035] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "46747e39d2fe422b983be3a6c560eee3" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 13:57:59.043972 23894 consensus_queue.cc:1035] T cddd86a956114ec6ad2a55ca1410035b P 3b0d1a154ac44337a0e2cf8a0f2f8e0d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250905 13:57:59.090034 23902 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:57:59.090596 23902 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:57:59.117782 23902 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20250905 13:57:59.297852 23887 consensus_queue.cc:1035] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b0d1a154ac44337a0e2cf8a0f2f8e0d" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 13:57:59.324064 23918 consensus_queue.cc:1035] T 69a6b46d98224a82992626a208f7642a P 46747e39d2fe422b983be3a6c560eee3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5aeb3545a65448fea2bf75704fd8a1ab" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 38791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20250905 13:58:00.407758 23902 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.255s user 0.493s sys 0.759s
W20250905 13:58:00.408032 23902 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.256s user 0.493s sys 0.759s
W20250905 13:58:01.781162 23928 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:01.781710 23928 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:01.809816 23928 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250905 13:58:03.031945 23928 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.185s user 0.450s sys 0.734s
W20250905 13:58:03.032373 23928 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.186s user 0.450s sys 0.734s
W20250905 13:58:04.396617 23943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:04.397109 23943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:04.424525 23943 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250905 13:58:05.621869 23943 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.162s user 0.418s sys 0.742s
W20250905 13:58:05.622197 23943 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.163s user 0.418s sys 0.742s
W20250905 13:58:07.044706 23958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:07.045306 23958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:07.074282 23958 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250905 13:58:08.266064 23958 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.155s user 0.434s sys 0.719s
W20250905 13:58:08.266479 23958 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.156s user 0.434s sys 0.719s
I20250905 13:58:09.341065 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 23466
I20250905 13:58:09.365708 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 23599
I20250905 13:58:09.388669 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 23732
I20250905 13:58:09.414835 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 23374
2025-09-05T13:58:09Z chronyd exiting
[ OK ] AdminCliTest.TestDescribeTableColumnFlags (18440 ms)
[ RUN ] AdminCliTest.TestAuthzResetCacheNotAuthorized
I20250905 13:58:09.468760 21899 test_util.cc:276] Using random seed: 978740545
I20250905 13:58:09.472384 21899 ts_itest-base.cc:115] Starting cluster with:
I20250905 13:58:09.472512 21899 ts_itest-base.cc:116] --------------
I20250905 13:58:09.472608 21899 ts_itest-base.cc:117] 3 tablet servers
I20250905 13:58:09.472698 21899 ts_itest-base.cc:118] 3 replicas per TS
I20250905 13:58:09.472800 21899 ts_itest-base.cc:119] --------------
2025-09-05T13:58:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T13:58:09Z Disabled control of system clock
I20250905 13:58:09.506042 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:36159
--webserver_interface=127.21.98.254
--webserver_port=0
--builtin_ntp_servers=127.21.98.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:36159
--superuser_acl=no-such-user with env {}
W20250905 13:58:09.790251 23980 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:09.790735 23980 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:09.791208 23980 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:09.819630 23980 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:58:09.819859 23980 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:09.820031 23980 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:58:09.820197 23980 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:58:09.851518 23980 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:36159
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:36159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:09.852679 23980 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:09.854090 23980 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:09.863719 23986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:11.267508 23985 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ec0 after lost signal to thread 23980
W20250905 13:58:11.564783 23980 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.701s user 0.611s sys 1.089s
W20250905 13:58:09.863996 23987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:11.565270 23980 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.701s user 0.611s sys 1.089s
W20250905 13:58:11.567409 23989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:11.569205 23988 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1700 milliseconds
I20250905 13:58:11.569227 23980 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:11.570317 23980 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:11.572489 23980 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:11.573793 23980 hybrid_clock.cc:648] HybridClock initialized: now 1757080691573766 us; error 27 us; skew 500 ppm
I20250905 13:58:11.574543 23980 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:11.580132 23980 webserver.cc:480] Webserver started at http://127.21.98.254:45963/ using document root <none> and password file <none>
I20250905 13:58:11.580955 23980 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:11.581142 23980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:11.581537 23980 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:58:11.585525 23980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "b69f334c672a4350898562c664eae502"
format_stamp: "Formatted at 2025-09-05 13:58:11 on dist-test-slave-g001"
I20250905 13:58:11.586512 23980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "b69f334c672a4350898562c664eae502"
format_stamp: "Formatted at 2025-09-05 13:58:11 on dist-test-slave-g001"
I20250905 13:58:11.593150 23980 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.001s
I20250905 13:58:11.598457 23996 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:11.599411 23980 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20250905 13:58:11.599736 23980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "b69f334c672a4350898562c664eae502"
format_stamp: "Formatted at 2025-09-05 13:58:11 on dist-test-slave-g001"
I20250905 13:58:11.600050 23980 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:11.650672 23980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:11.652009 23980 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:11.652428 23980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:11.718577 23980 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:36159
I20250905 13:58:11.718647 24047 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:36159 every 8 connection(s)
I20250905 13:58:11.721148 23980 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 13:58:11.726696 24048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:11.730057 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 23980
I20250905 13:58:11.730384 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 13:58:11.747960 24048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502: Bootstrap starting.
I20250905 13:58:11.752748 24048 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502: Neither blocks nor log segments found. Creating new log.
I20250905 13:58:11.754290 24048 log.cc:826] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:11.758550 24048 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502: No bootstrap required, opened a new log
I20250905 13:58:11.775560 24048 raft_consensus.cc:357] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69f334c672a4350898562c664eae502" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36159 } }
I20250905 13:58:11.776134 24048 raft_consensus.cc:383] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:58:11.776336 24048 raft_consensus.cc:738] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b69f334c672a4350898562c664eae502, State: Initialized, Role: FOLLOWER
I20250905 13:58:11.776829 24048 consensus_queue.cc:260] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69f334c672a4350898562c664eae502" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36159 } }
I20250905 13:58:11.777235 24048 raft_consensus.cc:397] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:11.777427 24048 raft_consensus.cc:491] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:11.777637 24048 raft_consensus.cc:3058] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:58:11.781018 24048 raft_consensus.cc:513] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69f334c672a4350898562c664eae502" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36159 } }
I20250905 13:58:11.781647 24048 leader_election.cc:304] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b69f334c672a4350898562c664eae502; no voters:
I20250905 13:58:11.783293 24048 leader_election.cc:290] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:58:11.783921 24053 raft_consensus.cc:2802] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:58:11.785959 24053 raft_consensus.cc:695] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [term 1 LEADER]: Becoming Leader. State: Replica: b69f334c672a4350898562c664eae502, State: Running, Role: LEADER
I20250905 13:58:11.786509 24053 consensus_queue.cc:237] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69f334c672a4350898562c664eae502" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36159 } }
I20250905 13:58:11.786859 24048 sys_catalog.cc:564] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:58:11.796142 24055 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b69f334c672a4350898562c664eae502. Latest consensus state: current_term: 1 leader_uuid: "b69f334c672a4350898562c664eae502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69f334c672a4350898562c664eae502" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36159 } } }
I20250905 13:58:11.796867 24055 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [sys.catalog]: This master's current role is: LEADER
I20250905 13:58:11.799392 24054 sys_catalog.cc:455] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b69f334c672a4350898562c664eae502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b69f334c672a4350898562c664eae502" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36159 } } }
I20250905 13:58:11.799937 24061 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:58:11.800127 24054 sys_catalog.cc:458] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502 [sys.catalog]: This master's current role is: LEADER
I20250905 13:58:11.809939 24061 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:58:11.823401 24061 catalog_manager.cc:1349] Generated new cluster ID: c5157b61ee9441fa8121771670f191a7
I20250905 13:58:11.823664 24061 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:58:11.847501 24061 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 13:58:11.848820 24061 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:58:11.862744 24061 catalog_manager.cc:5955] T 00000000000000000000000000000000 P b69f334c672a4350898562c664eae502: Generated new TSK 0
I20250905 13:58:11.863490 24061 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 13:58:11.882325 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:0
--local_ip_for_outbound_sockets=127.21.98.193
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36159
--builtin_ntp_servers=127.21.98.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250905 13:58:12.159792 24072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:12.160226 24072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:12.160696 24072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:12.189165 24072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:12.189893 24072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 13:58:12.220932 24072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:12.222256 24072 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:12.223733 24072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:12.235515 24078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:13.639668 24077 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 24072
W20250905 13:58:12.236943 24079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:13.663450 24072 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.427s user 0.423s sys 0.956s
W20250905 13:58:13.665807 24081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:13.665998 24072 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.430s user 0.423s sys 0.956s
W20250905 13:58:13.669250 24080 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1427 milliseconds
I20250905 13:58:13.669248 24072 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:13.670552 24072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:13.672544 24072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:13.673863 24072 hybrid_clock.cc:648] HybridClock initialized: now 1757080693673836 us; error 34 us; skew 500 ppm
I20250905 13:58:13.674598 24072 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:13.680769 24072 webserver.cc:480] Webserver started at http://127.21.98.193:44463/ using document root <none> and password file <none>
I20250905 13:58:13.681627 24072 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:13.681833 24072 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:13.682267 24072 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:58:13.686646 24072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "1954d91168034cde95cc2a413e9e55e0"
format_stamp: "Formatted at 2025-09-05 13:58:13 on dist-test-slave-g001"
I20250905 13:58:13.687780 24072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "1954d91168034cde95cc2a413e9e55e0"
format_stamp: "Formatted at 2025-09-05 13:58:13 on dist-test-slave-g001"
I20250905 13:58:13.695283 24072 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20250905 13:58:13.701035 24088 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:13.702003 24072 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250905 13:58:13.702282 24072 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "1954d91168034cde95cc2a413e9e55e0"
format_stamp: "Formatted at 2025-09-05 13:58:13 on dist-test-slave-g001"
I20250905 13:58:13.702561 24072 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:13.752816 24072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:13.754177 24072 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:13.754637 24072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:13.757663 24072 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:13.761442 24072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:58:13.761626 24072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:13.761883 24072 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:58:13.762034 24072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:13.928985 24072 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:41621
I20250905 13:58:13.929160 24200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:41621 every 8 connection(s)
I20250905 13:58:13.931529 24072 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 13:58:13.938612 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 24072
I20250905 13:58:13.939146 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 13:58:13.944633 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.194:0
--local_ip_for_outbound_sockets=127.21.98.194
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36159
--builtin_ntp_servers=127.21.98.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:58:13.956621 24201 heartbeater.cc:344] Connected to a master server at 127.21.98.254:36159
I20250905 13:58:13.957075 24201 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:13.958364 24201 heartbeater.cc:507] Master 127.21.98.254:36159 requested a full tablet report, sending...
I20250905 13:58:13.961136 24013 ts_manager.cc:194] Registered new tserver with Master: 1954d91168034cde95cc2a413e9e55e0 (127.21.98.193:41621)
I20250905 13:58:13.963624 24013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:36701
W20250905 13:58:14.233836 24205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:14.234352 24205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:14.234979 24205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:14.266058 24205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:14.266997 24205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.194
I20250905 13:58:14.299734 24205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.194
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:14.300969 24205 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:14.302547 24205 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:14.313237 24211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:58:14.967679 24201 heartbeater.cc:499] Master 127.21.98.254:36159 was elected leader, sending a full tablet report...
W20250905 13:58:14.315006 24212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:15.414127 24214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:15.415253 24213 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1096 milliseconds
I20250905 13:58:15.415390 24205 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:15.416501 24205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:15.418438 24205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:15.419764 24205 hybrid_clock.cc:648] HybridClock initialized: now 1757080695419727 us; error 43 us; skew 500 ppm
I20250905 13:58:15.420491 24205 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:15.425769 24205 webserver.cc:480] Webserver started at http://127.21.98.194:45081/ using document root <none> and password file <none>
I20250905 13:58:15.426620 24205 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:15.426821 24205 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:15.427271 24205 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:58:15.431125 24205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "89be351c6c3447ae9c6bdf52c9540a98"
format_stamp: "Formatted at 2025-09-05 13:58:15 on dist-test-slave-g001"
I20250905 13:58:15.432173 24205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "89be351c6c3447ae9c6bdf52c9540a98"
format_stamp: "Formatted at 2025-09-05 13:58:15 on dist-test-slave-g001"
I20250905 13:58:15.438738 24205 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250905 13:58:15.443677 24221 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:15.444496 24205 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20250905 13:58:15.444805 24205 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "89be351c6c3447ae9c6bdf52c9540a98"
format_stamp: "Formatted at 2025-09-05 13:58:15 on dist-test-slave-g001"
I20250905 13:58:15.445099 24205 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:15.505919 24205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:15.507156 24205 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:15.507531 24205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:15.509776 24205 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:15.513376 24205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:58:15.513553 24205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:15.513767 24205 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:58:15.513888 24205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:15.639163 24205 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.194:33811
I20250905 13:58:15.639248 24333 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.194:33811 every 8 connection(s)
I20250905 13:58:15.641669 24205 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 13:58:15.644238 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 24205
I20250905 13:58:15.644888 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 13:58:15.651011 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.195:0
--local_ip_for_outbound_sockets=127.21.98.195
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36159
--builtin_ntp_servers=127.21.98.212:38385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:58:15.661515 24334 heartbeater.cc:344] Connected to a master server at 127.21.98.254:36159
I20250905 13:58:15.661880 24334 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:15.662760 24334 heartbeater.cc:507] Master 127.21.98.254:36159 requested a full tablet report, sending...
I20250905 13:58:15.664709 24013 ts_manager.cc:194] Registered new tserver with Master: 89be351c6c3447ae9c6bdf52c9540a98 (127.21.98.194:33811)
I20250905 13:58:15.665900 24013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.194:44749
W20250905 13:58:15.948567 24338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:15.948979 24338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:15.949451 24338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:15.977934 24338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:15.978665 24338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.195
I20250905 13:58:16.010388 24338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:38385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36159
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.195
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:16.011619 24338 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:16.013151 24338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:16.024236 24344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:58:16.668742 24334 heartbeater.cc:499] Master 127.21.98.254:36159 was elected leader, sending a full tablet report...
W20250905 13:58:16.025141 24345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:17.306744 24347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:17.310556 24346 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1284 milliseconds
W20250905 13:58:17.311018 24338 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.287s user 0.438s sys 0.825s
W20250905 13:58:17.311301 24338 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.287s user 0.442s sys 0.825s
I20250905 13:58:17.311468 24338 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:17.312417 24338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:17.319736 24338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:17.321192 24338 hybrid_clock.cc:648] HybridClock initialized: now 1757080697321138 us; error 52 us; skew 500 ppm
I20250905 13:58:17.322049 24338 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:17.328480 24338 webserver.cc:480] Webserver started at http://127.21.98.195:35189/ using document root <none> and password file <none>
I20250905 13:58:17.329344 24338 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:17.329566 24338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:17.329964 24338 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:58:17.333923 24338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a92986b224544017bbf2864dff705b0c"
format_stamp: "Formatted at 2025-09-05 13:58:17 on dist-test-slave-g001"
I20250905 13:58:17.335068 24338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a92986b224544017bbf2864dff705b0c"
format_stamp: "Formatted at 2025-09-05 13:58:17 on dist-test-slave-g001"
I20250905 13:58:17.342298 24338 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20250905 13:58:17.347988 24354 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:17.349046 24338 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20250905 13:58:17.349344 24338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a92986b224544017bbf2864dff705b0c"
format_stamp: "Formatted at 2025-09-05 13:58:17 on dist-test-slave-g001"
I20250905 13:58:17.349625 24338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:17.411597 24338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:17.413010 24338 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:17.413494 24338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:17.415931 24338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:17.419693 24338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:58:17.419878 24338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:17.420064 24338 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:58:17.420190 24338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:17.547541 24338 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.195:42213
I20250905 13:58:17.547634 24466 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.195:42213 every 8 connection(s)
I20250905 13:58:17.549957 24338 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 13:58:17.554267 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 24338
I20250905 13:58:17.554764 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestAuthzResetCacheNotAuthorized.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 13:58:17.573436 24467 heartbeater.cc:344] Connected to a master server at 127.21.98.254:36159
I20250905 13:58:17.573804 24467 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:17.574684 24467 heartbeater.cc:507] Master 127.21.98.254:36159 requested a full tablet report, sending...
I20250905 13:58:17.576728 24013 ts_manager.cc:194] Registered new tserver with Master: a92986b224544017bbf2864dff705b0c (127.21.98.195:42213)
I20250905 13:58:17.577844 24013 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.195:42781
I20250905 13:58:17.587275 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:58:17.621908 24013 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45656:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
W20250905 13:58:17.639268 24013 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table TestTable in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 13:58:17.687438 24269 tablet_service.cc:1468] Processing CreateTablet for tablet a50453c14e664cdc802ca746291851f5 (DEFAULT_TABLE table=TestTable [id=5dbaaff4210946aa950e2b7437ac6833]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:58:17.688217 24136 tablet_service.cc:1468] Processing CreateTablet for tablet a50453c14e664cdc802ca746291851f5 (DEFAULT_TABLE table=TestTable [id=5dbaaff4210946aa950e2b7437ac6833]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:58:17.688898 24269 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50453c14e664cdc802ca746291851f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:17.689936 24136 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50453c14e664cdc802ca746291851f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:17.689308 24402 tablet_service.cc:1468] Processing CreateTablet for tablet a50453c14e664cdc802ca746291851f5 (DEFAULT_TABLE table=TestTable [id=5dbaaff4210946aa950e2b7437ac6833]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:58:17.690985 24402 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a50453c14e664cdc802ca746291851f5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:17.708189 24487 tablet_bootstrap.cc:492] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98: Bootstrap starting.
I20250905 13:58:17.714331 24487 tablet_bootstrap.cc:654] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98: Neither blocks nor log segments found. Creating new log.
I20250905 13:58:17.715576 24488 tablet_bootstrap.cc:492] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0: Bootstrap starting.
I20250905 13:58:17.716727 24487 log.cc:826] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:17.722895 24488 tablet_bootstrap.cc:654] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0: Neither blocks nor log segments found. Creating new log.
I20250905 13:58:17.722787 24489 tablet_bootstrap.cc:492] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c: Bootstrap starting.
I20250905 13:58:17.725919 24488 log.cc:826] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:17.729966 24487 tablet_bootstrap.cc:492] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98: No bootstrap required, opened a new log
I20250905 13:58:17.730494 24487 ts_tablet_manager.cc:1397] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.003s
I20250905 13:58:17.730286 24489 tablet_bootstrap.cc:654] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c: Neither blocks nor log segments found. Creating new log.
I20250905 13:58:17.733083 24489 log.cc:826] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:17.736949 24488 tablet_bootstrap.cc:492] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0: No bootstrap required, opened a new log
I20250905 13:58:17.737429 24488 ts_tablet_manager.cc:1397] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.010s
I20250905 13:58:17.758206 24489 tablet_bootstrap.cc:492] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c: No bootstrap required, opened a new log
I20250905 13:58:17.758729 24489 ts_tablet_manager.cc:1397] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c: Time spent bootstrapping tablet: real 0.036s user 0.012s sys 0.021s
I20250905 13:58:17.762370 24487 raft_consensus.cc:357] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1954d91168034cde95cc2a413e9e55e0" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41621 } } peers { permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 } } peers { permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 } }
I20250905 13:58:17.763428 24487 raft_consensus.cc:383] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:58:17.763775 24487 raft_consensus.cc:738] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89be351c6c3447ae9c6bdf52c9540a98, State: Initialized, Role: FOLLOWER
I20250905 13:58:17.763429 24488 raft_consensus.cc:357] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1954d91168034cde95cc2a413e9e55e0" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41621 } } peers { permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 } } peers { permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 } }
I20250905 13:58:17.765542 24487 consensus_queue.cc:260] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1954d91168034cde95cc2a413e9e55e0" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41621 } } peers { permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 } } peers { permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 } }
I20250905 13:58:17.764359 24488 raft_consensus.cc:383] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:58:17.767189 24488 raft_consensus.cc:738] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1954d91168034cde95cc2a413e9e55e0, State: Initialized, Role: FOLLOWER
I20250905 13:58:17.768407 24488 consensus_queue.cc:260] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1954d91168034cde95cc2a413e9e55e0" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41621 } } peers { permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 } } peers { permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 } }
I20250905 13:58:17.779129 24488 ts_tablet_manager.cc:1428] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0: Time spent starting tablet: real 0.041s user 0.022s sys 0.013s
I20250905 13:58:17.781863 24487 ts_tablet_manager.cc:1428] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98: Time spent starting tablet: real 0.051s user 0.025s sys 0.013s
I20250905 13:58:17.784272 24489 raft_consensus.cc:357] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1954d91168034cde95cc2a413e9e55e0" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41621 } } peers { permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 } } peers { permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 } }
I20250905 13:58:17.785176 24489 raft_consensus.cc:383] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:58:17.785533 24489 raft_consensus.cc:738] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a92986b224544017bbf2864dff705b0c, State: Initialized, Role: FOLLOWER
I20250905 13:58:17.786584 24489 consensus_queue.cc:260] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1954d91168034cde95cc2a413e9e55e0" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41621 } } peers { permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 } } peers { permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 } }
I20250905 13:58:17.791107 24467 heartbeater.cc:499] Master 127.21.98.254:36159 was elected leader, sending a full tablet report...
I20250905 13:58:17.791924 24489 ts_tablet_manager.cc:1428] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c: Time spent starting tablet: real 0.033s user 0.028s sys 0.003s
W20250905 13:58:17.804092 24468 tablet.cc:2378] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 13:58:17.898072 24335 tablet.cc:2378] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 13:58:17.940156 24202 tablet.cc:2378] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:58:17.968993 24494 raft_consensus.cc:491] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:58:17.969403 24494 raft_consensus.cc:513] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1954d91168034cde95cc2a413e9e55e0" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41621 } } peers { permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 } } peers { permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 } }
I20250905 13:58:17.972158 24494 leader_election.cc:290] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a92986b224544017bbf2864dff705b0c (127.21.98.195:42213), 89be351c6c3447ae9c6bdf52c9540a98 (127.21.98.194:33811)
I20250905 13:58:17.982259 24289 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a50453c14e664cdc802ca746291851f5" candidate_uuid: "1954d91168034cde95cc2a413e9e55e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89be351c6c3447ae9c6bdf52c9540a98" is_pre_election: true
I20250905 13:58:17.982299 24422 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a50453c14e664cdc802ca746291851f5" candidate_uuid: "1954d91168034cde95cc2a413e9e55e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a92986b224544017bbf2864dff705b0c" is_pre_election: true
I20250905 13:58:17.982865 24422 raft_consensus.cc:2466] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1954d91168034cde95cc2a413e9e55e0 in term 0.
I20250905 13:58:17.982856 24289 raft_consensus.cc:2466] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1954d91168034cde95cc2a413e9e55e0 in term 0.
I20250905 13:58:17.983837 24089 leader_election.cc:304] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1954d91168034cde95cc2a413e9e55e0, a92986b224544017bbf2864dff705b0c; no voters:
I20250905 13:58:17.984485 24494 raft_consensus.cc:2802] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 13:58:17.984733 24494 raft_consensus.cc:491] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 13:58:17.984956 24494 raft_consensus.cc:3058] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:58:17.989053 24494 raft_consensus.cc:513] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1954d91168034cde95cc2a413e9e55e0" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41621 } } peers { permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 } } peers { permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 } }
I20250905 13:58:17.990145 24494 leader_election.cc:290] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [CANDIDATE]: Term 1 election: Requested vote from peers a92986b224544017bbf2864dff705b0c (127.21.98.195:42213), 89be351c6c3447ae9c6bdf52c9540a98 (127.21.98.194:33811)
I20250905 13:58:17.990739 24422 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a50453c14e664cdc802ca746291851f5" candidate_uuid: "1954d91168034cde95cc2a413e9e55e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a92986b224544017bbf2864dff705b0c"
I20250905 13:58:17.991036 24289 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a50453c14e664cdc802ca746291851f5" candidate_uuid: "1954d91168034cde95cc2a413e9e55e0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "89be351c6c3447ae9c6bdf52c9540a98"
I20250905 13:58:17.991250 24422 raft_consensus.cc:3058] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:58:17.991532 24289 raft_consensus.cc:3058] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:58:17.997931 24289 raft_consensus.cc:2466] T a50453c14e664cdc802ca746291851f5 P 89be351c6c3447ae9c6bdf52c9540a98 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1954d91168034cde95cc2a413e9e55e0 in term 1.
I20250905 13:58:17.997931 24422 raft_consensus.cc:2466] T a50453c14e664cdc802ca746291851f5 P a92986b224544017bbf2864dff705b0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1954d91168034cde95cc2a413e9e55e0 in term 1.
I20250905 13:58:17.998862 24092 leader_election.cc:304] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1954d91168034cde95cc2a413e9e55e0, 89be351c6c3447ae9c6bdf52c9540a98; no voters:
I20250905 13:58:17.999508 24494 raft_consensus.cc:2802] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:58:18.000911 24494 raft_consensus.cc:695] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [term 1 LEADER]: Becoming Leader. State: Replica: 1954d91168034cde95cc2a413e9e55e0, State: Running, Role: LEADER
I20250905 13:58:18.001763 24494 consensus_queue.cc:237] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1954d91168034cde95cc2a413e9e55e0" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41621 } } peers { permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 } } peers { permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 } }
I20250905 13:58:18.012130 24011 catalog_manager.cc:5582] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1954d91168034cde95cc2a413e9e55e0 (127.21.98.193). New cstate: current_term: 1 leader_uuid: "1954d91168034cde95cc2a413e9e55e0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1954d91168034cde95cc2a413e9e55e0" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 41621 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 } health_report { overall_health: UNKNOWN } } }
I20250905 13:58:18.051374 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:58:18.054289 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 1954d91168034cde95cc2a413e9e55e0 to finish bootstrapping
I20250905 13:58:18.066208 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 89be351c6c3447ae9c6bdf52c9540a98 to finish bootstrapping
I20250905 13:58:18.076431 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver a92986b224544017bbf2864dff705b0c to finish bootstrapping
I20250905 13:58:18.390004 24494 consensus_queue.cc:1035] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "89be351c6c3447ae9c6bdf52c9540a98" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 33811 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 13:58:18.405956 24494 consensus_queue.cc:1035] T a50453c14e664cdc802ca746291851f5 P 1954d91168034cde95cc2a413e9e55e0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a92986b224544017bbf2864dff705b0c" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 42213 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 13:58:19.597968 24011 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.RefreshAuthzCache from {username='slave'} at 127.0.0.1:45670
I20250905 13:58:20.626742 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 24072
I20250905 13:58:20.656877 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 24205
I20250905 13:58:20.683424 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 24338
I20250905 13:58:20.705749 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 23980
2025-09-05T13:58:20Z chronyd exiting
[ OK ] AdminCliTest.TestAuthzResetCacheNotAuthorized (11283 ms)
[ RUN ] AdminCliTest.TestRebuildTables
I20250905 13:58:20.752633 21899 test_util.cc:276] Using random seed: 990024416
I20250905 13:58:20.756459 21899 ts_itest-base.cc:115] Starting cluster with:
I20250905 13:58:20.756585 21899 ts_itest-base.cc:116] --------------
I20250905 13:58:20.756731 21899 ts_itest-base.cc:117] 3 tablet servers
I20250905 13:58:20.756853 21899 ts_itest-base.cc:118] 3 replicas per TS
I20250905 13:58:20.756973 21899 ts_itest-base.cc:119] --------------
2025-09-05T13:58:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T13:58:20Z Disabled control of system clock
I20250905 13:58:20.790174 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:41873
--webserver_interface=127.21.98.254
--webserver_port=0
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:41873 with env {}
W20250905 13:58:21.055174 24537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:21.055716 24537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:21.056164 24537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:21.083727 24537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:58:21.083959 24537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:21.084218 24537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:58:21.084435 24537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:58:21.114051 24537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:41873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:21.115248 24537 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:21.116755 24537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:21.125717 24543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:21.127743 24544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:22.332974 24545 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1201 milliseconds
W20250905 13:58:22.334909 24546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:22.337131 24537 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.210s user 0.000s sys 0.007s
W20250905 13:58:22.337368 24537 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.211s user 0.000s sys 0.007s
I20250905 13:58:22.337587 24537 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:22.338575 24537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:22.340867 24537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:22.342182 24537 hybrid_clock.cc:648] HybridClock initialized: now 1757080702342151 us; error 30 us; skew 500 ppm
I20250905 13:58:22.342953 24537 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:22.349972 24537 webserver.cc:480] Webserver started at http://127.21.98.254:42307/ using document root <none> and password file <none>
I20250905 13:58:22.350865 24537 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:22.351121 24537 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:22.351553 24537 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:58:22.355795 24537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "705a9105a9124c8e93599291af08d21a"
format_stamp: "Formatted at 2025-09-05 13:58:22 on dist-test-slave-g001"
I20250905 13:58:22.356804 24537 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "705a9105a9124c8e93599291af08d21a"
format_stamp: "Formatted at 2025-09-05 13:58:22 on dist-test-slave-g001"
I20250905 13:58:22.364084 24537 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250905 13:58:22.369488 24553 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:22.370467 24537 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 13:58:22.370790 24537 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "705a9105a9124c8e93599291af08d21a"
format_stamp: "Formatted at 2025-09-05 13:58:22 on dist-test-slave-g001"
I20250905 13:58:22.371162 24537 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:22.451680 24537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:22.453028 24537 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:22.453416 24537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:22.516445 24537 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:41873
I20250905 13:58:22.516513 24604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:41873 every 8 connection(s)
I20250905 13:58:22.518916 24537 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 13:58:22.523967 24605 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:22.527813 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 24537
I20250905 13:58:22.528136 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 13:58:22.544350 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Bootstrap starting.
I20250905 13:58:22.549589 24605 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Neither blocks nor log segments found. Creating new log.
I20250905 13:58:22.551149 24605 log.cc:826] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:22.555140 24605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: No bootstrap required, opened a new log
I20250905 13:58:22.569926 24605 raft_consensus.cc:357] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:22.570454 24605 raft_consensus.cc:383] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:58:22.570678 24605 raft_consensus.cc:738] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705a9105a9124c8e93599291af08d21a, State: Initialized, Role: FOLLOWER
I20250905 13:58:22.571270 24605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:22.571729 24605 raft_consensus.cc:397] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:22.571949 24605 raft_consensus.cc:491] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:22.572201 24605 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:58:22.576134 24605 raft_consensus.cc:513] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:22.576719 24605 leader_election.cc:304] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 705a9105a9124c8e93599291af08d21a; no voters:
I20250905 13:58:22.578156 24605 leader_election.cc:290] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:58:22.578842 24610 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:58:22.580919 24610 raft_consensus.cc:695] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 1 LEADER]: Becoming Leader. State: Replica: 705a9105a9124c8e93599291af08d21a, State: Running, Role: LEADER
I20250905 13:58:22.581627 24610 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:22.582410 24605 sys_catalog.cc:564] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:58:22.587693 24611 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "705a9105a9124c8e93599291af08d21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } } }
I20250905 13:58:22.588425 24611 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: This master's current role is: LEADER
I20250905 13:58:22.589336 24612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 705a9105a9124c8e93599291af08d21a. Latest consensus state: current_term: 1 leader_uuid: "705a9105a9124c8e93599291af08d21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } } }
I20250905 13:58:22.591622 24612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: This master's current role is: LEADER
I20250905 13:58:22.593677 24619 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:58:22.603174 24619 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:58:22.617354 24619 catalog_manager.cc:1349] Generated new cluster ID: 16f5536e4a9e4d689dd4678a1c856d3c
I20250905 13:58:22.617626 24619 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:58:22.643440 24619 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 13:58:22.645018 24619 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:58:22.672266 24619 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Generated new TSK 0
I20250905 13:58:22.673151 24619 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 13:58:22.687882 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:0
--local_ip_for_outbound_sockets=127.21.98.193
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:41873
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250905 13:58:22.976868 24629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:22.977313 24629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:22.977766 24629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:23.006541 24629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:23.007282 24629 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 13:58:23.038328 24629 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:41873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:23.039559 24629 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:23.041138 24629 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:23.053135 24635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:23.053470 24636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:24.154422 24638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:24.158401 24629 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.105s user 0.366s sys 0.735s
W20250905 13:58:24.158758 24629 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.106s user 0.366s sys 0.735s
I20250905 13:58:24.159047 24629 server_base.cc:1047] running on GCE node
I20250905 13:58:24.160336 24629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:24.163038 24629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:24.164573 24629 hybrid_clock.cc:648] HybridClock initialized: now 1757080704164492 us; error 74 us; skew 500 ppm
I20250905 13:58:24.165679 24629 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:24.174574 24629 webserver.cc:480] Webserver started at http://127.21.98.193:36177/ using document root <none> and password file <none>
I20250905 13:58:24.175851 24629 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:24.176118 24629 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:24.176765 24629 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:58:24.183434 24629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "b41ea60da9974dabb2e4bb44807a7c9f"
format_stamp: "Formatted at 2025-09-05 13:58:24 on dist-test-slave-g001"
I20250905 13:58:24.184976 24629 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "b41ea60da9974dabb2e4bb44807a7c9f"
format_stamp: "Formatted at 2025-09-05 13:58:24 on dist-test-slave-g001"
I20250905 13:58:24.195127 24629 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.009s sys 0.001s
I20250905 13:58:24.203938 24645 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:24.205205 24629 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20250905 13:58:24.205619 24629 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b41ea60da9974dabb2e4bb44807a7c9f"
format_stamp: "Formatted at 2025-09-05 13:58:24 on dist-test-slave-g001"
I20250905 13:58:24.206036 24629 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:24.279502 24629 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:24.281268 24629 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:24.281744 24629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:24.284822 24629 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:24.290431 24629 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:58:24.290709 24629 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:24.291008 24629 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:58:24.291208 24629 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:24.432653 24629 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:35763
I20250905 13:58:24.432708 24757 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:35763 every 8 connection(s)
I20250905 13:58:24.436259 24629 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 13:58:24.438910 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 24629
I20250905 13:58:24.439352 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 13:58:24.449585 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.194:0
--local_ip_for_outbound_sockets=127.21.98.194
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:41873
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:58:24.484848 24758 heartbeater.cc:344] Connected to a master server at 127.21.98.254:41873
I20250905 13:58:24.485365 24758 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:24.486662 24758 heartbeater.cc:507] Master 127.21.98.254:41873 requested a full tablet report, sending...
I20250905 13:58:24.489495 24570 ts_manager.cc:194] Registered new tserver with Master: b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193:35763)
I20250905 13:58:24.492133 24570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:52079
W20250905 13:58:24.737512 24762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:24.737936 24762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:24.738366 24762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:24.765834 24762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:24.766665 24762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.194
I20250905 13:58:24.797662 24762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:41873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.194
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:24.798857 24762 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:24.800267 24762 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:24.811295 24768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:58:25.495774 24758 heartbeater.cc:499] Master 127.21.98.254:41873 was elected leader, sending a full tablet report...
W20250905 13:58:26.214359 24767 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 24762
W20250905 13:58:26.409889 24767 kernel_stack_watchdog.cc:198] Thread 24762 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 13:58:24.811877 24769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:26.411437 24762 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.600s user 0.653s sys 0.931s
W20250905 13:58:26.411886 24762 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.600s user 0.653s sys 0.931s
W20250905 13:58:26.413551 24771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:26.415582 24770 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1598 milliseconds
I20250905 13:58:26.415614 24762 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:26.416777 24762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:26.418948 24762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:26.420323 24762 hybrid_clock.cc:648] HybridClock initialized: now 1757080706420271 us; error 52 us; skew 500 ppm
I20250905 13:58:26.421116 24762 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:26.426852 24762 webserver.cc:480] Webserver started at http://127.21.98.194:41835/ using document root <none> and password file <none>
I20250905 13:58:26.427795 24762 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:26.428004 24762 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:26.428486 24762 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:58:26.432739 24762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "c656231d41e2493f955684515def253f"
format_stamp: "Formatted at 2025-09-05 13:58:26 on dist-test-slave-g001"
I20250905 13:58:26.433804 24762 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "c656231d41e2493f955684515def253f"
format_stamp: "Formatted at 2025-09-05 13:58:26 on dist-test-slave-g001"
I20250905 13:58:26.441079 24762 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20250905 13:58:26.446493 24778 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:26.447609 24762 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20250905 13:58:26.447927 24762 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c656231d41e2493f955684515def253f"
format_stamp: "Formatted at 2025-09-05 13:58:26 on dist-test-slave-g001"
I20250905 13:58:26.448259 24762 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:26.519672 24762 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:26.521406 24762 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:26.521890 24762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:26.524770 24762 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:26.529973 24762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:58:26.530210 24762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:26.530524 24762 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:58:26.530728 24762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:26.656545 24762 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.194:39317
I20250905 13:58:26.656662 24890 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.194:39317 every 8 connection(s)
I20250905 13:58:26.659212 24762 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 13:58:26.668812 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 24762
I20250905 13:58:26.669137 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 13:58:26.675084 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.195:0
--local_ip_for_outbound_sockets=127.21.98.195
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:41873
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:58:26.679193 24891 heartbeater.cc:344] Connected to a master server at 127.21.98.254:41873
I20250905 13:58:26.679570 24891 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:26.680518 24891 heartbeater.cc:507] Master 127.21.98.254:41873 requested a full tablet report, sending...
I20250905 13:58:26.682730 24570 ts_manager.cc:194] Registered new tserver with Master: c656231d41e2493f955684515def253f (127.21.98.194:39317)
I20250905 13:58:26.684154 24570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.194:55757
W20250905 13:58:26.948858 24895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:26.949326 24895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:26.949805 24895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:26.978339 24895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:26.979140 24895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.195
I20250905 13:58:27.010118 24895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:41873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.195
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:27.011281 24895 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:27.012651 24895 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:27.023514 24901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:58:27.687512 24891 heartbeater.cc:499] Master 127.21.98.254:41873 was elected leader, sending a full tablet report...
W20250905 13:58:27.024255 24902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:28.427412 24900 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 24895
W20250905 13:58:28.656737 24895 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.633s user 0.624s sys 1.005s
W20250905 13:58:28.658643 24895 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.635s user 0.625s sys 1.005s
W20250905 13:58:28.658880 24904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:28.662271 24903 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1632 milliseconds
I20250905 13:58:28.662329 24895 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:28.663678 24895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:28.665727 24895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:28.667050 24895 hybrid_clock.cc:648] HybridClock initialized: now 1757080708667026 us; error 31 us; skew 500 ppm
I20250905 13:58:28.667774 24895 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:28.673484 24895 webserver.cc:480] Webserver started at http://127.21.98.195:36707/ using document root <none> and password file <none>
I20250905 13:58:28.674413 24895 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:28.674638 24895 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:28.675082 24895 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:58:28.679358 24895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "867e2a35e49a4320b858ce1efad4e408"
format_stamp: "Formatted at 2025-09-05 13:58:28 on dist-test-slave-g001"
I20250905 13:58:28.680333 24895 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "867e2a35e49a4320b858ce1efad4e408"
format_stamp: "Formatted at 2025-09-05 13:58:28 on dist-test-slave-g001"
I20250905 13:58:28.687000 24895 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20250905 13:58:28.692333 24912 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:28.693202 24895 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.003s
I20250905 13:58:28.693490 24895 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "867e2a35e49a4320b858ce1efad4e408"
format_stamp: "Formatted at 2025-09-05 13:58:28 on dist-test-slave-g001"
I20250905 13:58:28.693797 24895 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:28.747278 24895 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:28.748572 24895 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:28.748998 24895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:28.751276 24895 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:28.754876 24895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:58:28.755064 24895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:28.755322 24895 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:58:28.755488 24895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:28.884994 24895 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.195:43931
I20250905 13:58:28.885078 25024 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.195:43931 every 8 connection(s)
I20250905 13:58:28.887723 24895 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 13:58:28.893203 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 24895
I20250905 13:58:28.893750 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 13:58:28.908017 25025 heartbeater.cc:344] Connected to a master server at 127.21.98.254:41873
I20250905 13:58:28.908368 25025 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:28.909233 25025 heartbeater.cc:507] Master 127.21.98.254:41873 requested a full tablet report, sending...
I20250905 13:58:28.911381 24570 ts_manager.cc:194] Registered new tserver with Master: 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931)
I20250905 13:58:28.912670 24570 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.195:43995
I20250905 13:58:28.914577 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:58:28.948417 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:58:28.948712 21899 test_util.cc:276] Using random seed: 998220504
I20250905 13:58:28.985239 24569 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43416:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 13:58:29.022428 24826 tablet_service.cc:1468] Processing CreateTablet for tablet 2ef4f3179cbd4d71a473c89311a50045 (DEFAULT_TABLE table=TestTable [id=563c4b8f1723493283882b41d8dfbc7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:58:29.023743 24826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef4f3179cbd4d71a473c89311a50045. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:29.039955 25045 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Bootstrap starting.
I20250905 13:58:29.044792 25045 tablet_bootstrap.cc:654] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Neither blocks nor log segments found. Creating new log.
I20250905 13:58:29.046307 25045 log.cc:826] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:29.050244 25045 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: No bootstrap required, opened a new log
I20250905 13:58:29.050634 25045 ts_tablet_manager.cc:1397] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250905 13:58:29.065136 25045 raft_consensus.cc:357] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } }
I20250905 13:58:29.065605 25045 raft_consensus.cc:383] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:58:29.065872 25045 raft_consensus.cc:738] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c656231d41e2493f955684515def253f, State: Initialized, Role: FOLLOWER
I20250905 13:58:29.066496 25045 consensus_queue.cc:260] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } }
I20250905 13:58:29.067019 25045 raft_consensus.cc:397] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:29.067271 25045 raft_consensus.cc:491] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:29.067553 25045 raft_consensus.cc:3058] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:58:29.071122 25045 raft_consensus.cc:513] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } }
I20250905 13:58:29.071806 25045 leader_election.cc:304] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c656231d41e2493f955684515def253f; no voters:
I20250905 13:58:29.073326 25045 leader_election.cc:290] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:58:29.073770 25047 raft_consensus.cc:2802] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:58:29.076381 25047 raft_consensus.cc:695] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 1 LEADER]: Becoming Leader. State: Replica: c656231d41e2493f955684515def253f, State: Running, Role: LEADER
I20250905 13:58:29.077188 25047 consensus_queue.cc:237] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } }
I20250905 13:58:29.077620 25045 ts_tablet_manager.cc:1428] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Time spent starting tablet: real 0.027s user 0.016s sys 0.011s
I20250905 13:58:29.087714 24569 catalog_manager.cc:5582] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f reported cstate change: term changed from 0 to 1, leader changed from <none> to c656231d41e2493f955684515def253f (127.21.98.194). New cstate: current_term: 1 leader_uuid: "c656231d41e2493f955684515def253f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } health_report { overall_health: HEALTHY } } }
I20250905 13:58:29.247474 21899 test_util.cc:276] Using random seed: 998519257
I20250905 13:58:29.267331 24569 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43426:
name: "TestTable1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 13:58:29.293671 24960 tablet_service.cc:1468] Processing CreateTablet for tablet b8ff2659b4504a0b827dbdd7041eaf50 (DEFAULT_TABLE table=TestTable1 [id=f265f4053bcc4e989f3c5832401b9e8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:58:29.294986 24960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ff2659b4504a0b827dbdd7041eaf50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:29.312803 25066 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap starting.
I20250905 13:58:29.318223 25066 tablet_bootstrap.cc:654] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Neither blocks nor log segments found. Creating new log.
I20250905 13:58:29.319941 25066 log.cc:826] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:29.324245 25066 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: No bootstrap required, opened a new log
I20250905 13:58:29.324616 25066 ts_tablet_manager.cc:1397] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20250905 13:58:29.340991 25066 raft_consensus.cc:357] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } }
I20250905 13:58:29.341516 25066 raft_consensus.cc:383] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:58:29.341704 25066 raft_consensus.cc:738] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867e2a35e49a4320b858ce1efad4e408, State: Initialized, Role: FOLLOWER
I20250905 13:58:29.342332 25066 consensus_queue.cc:260] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } }
I20250905 13:58:29.342815 25066 raft_consensus.cc:397] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:29.343108 25066 raft_consensus.cc:491] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:29.343370 25066 raft_consensus.cc:3058] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:58:29.347223 25066 raft_consensus.cc:513] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } }
I20250905 13:58:29.347771 25066 leader_election.cc:304] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 867e2a35e49a4320b858ce1efad4e408; no voters:
I20250905 13:58:29.349463 25066 leader_election.cc:290] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:58:29.349740 25068 raft_consensus.cc:2802] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:58:29.352663 25025 heartbeater.cc:499] Master 127.21.98.254:41873 was elected leader, sending a full tablet report...
I20250905 13:58:29.352663 25068 raft_consensus.cc:695] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 1 LEADER]: Becoming Leader. State: Replica: 867e2a35e49a4320b858ce1efad4e408, State: Running, Role: LEADER
I20250905 13:58:29.352891 25066 ts_tablet_manager.cc:1428] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Time spent starting tablet: real 0.028s user 0.023s sys 0.005s
I20250905 13:58:29.353534 25068 consensus_queue.cc:237] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } }
I20250905 13:58:29.362663 24569 catalog_manager.cc:5582] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 reported cstate change: term changed from 0 to 1, leader changed from <none> to 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195). New cstate: current_term: 1 leader_uuid: "867e2a35e49a4320b858ce1efad4e408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } health_report { overall_health: HEALTHY } } }
I20250905 13:58:29.528746 21899 test_util.cc:276] Using random seed: 998800528
I20250905 13:58:29.554132 24570 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43428:
name: "TestTable2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 13:58:29.582952 24693 tablet_service.cc:1468] Processing CreateTablet for tablet 7c2039b8c6ed49eca474f06b27f83782 (DEFAULT_TABLE table=TestTable2 [id=f8721b6cc75b452da9ca09f60e16b422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:58:29.584431 24693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c2039b8c6ed49eca474f06b27f83782. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:29.601095 25087 tablet_bootstrap.cc:492] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap starting.
I20250905 13:58:29.606369 25087 tablet_bootstrap.cc:654] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Neither blocks nor log segments found. Creating new log.
I20250905 13:58:29.608016 25087 log.cc:826] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:29.611869 25087 tablet_bootstrap.cc:492] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: No bootstrap required, opened a new log
I20250905 13:58:29.612200 25087 ts_tablet_manager.cc:1397] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20250905 13:58:29.628492 25087 raft_consensus.cc:357] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:29.629037 25087 raft_consensus.cc:383] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:58:29.629225 25087 raft_consensus.cc:738] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Initialized, Role: FOLLOWER
I20250905 13:58:29.629850 25087 consensus_queue.cc:260] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:29.630282 25087 raft_consensus.cc:397] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:29.630527 25087 raft_consensus.cc:491] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:29.630846 25087 raft_consensus.cc:3058] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:58:29.634539 25087 raft_consensus.cc:513] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:29.635092 25087 leader_election.cc:304] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b41ea60da9974dabb2e4bb44807a7c9f; no voters:
I20250905 13:58:29.636799 25087 leader_election.cc:290] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:58:29.637544 25089 raft_consensus.cc:2802] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:58:29.640202 25087 ts_tablet_manager.cc:1428] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent starting tablet: real 0.028s user 0.021s sys 0.007s
I20250905 13:58:29.640128 25089 raft_consensus.cc:695] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 1 LEADER]: Becoming Leader. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Running, Role: LEADER
I20250905 13:58:29.640802 25089 consensus_queue.cc:237] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:29.648636 24570 catalog_manager.cc:5582] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f reported cstate change: term changed from 0 to 1, leader changed from <none> to b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193). New cstate: current_term: 1 leader_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } health_report { overall_health: HEALTHY } } }
I20250905 13:58:29.789367 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 24537
W20250905 13:58:30.101281 24891 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:41873: connect: Connection refused (error 111)
W20250905 13:58:30.382752 25025 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:41873: connect: Connection refused (error 111)
W20250905 13:58:30.680289 24758 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:41873: connect: Connection refused (error 111)
I20250905 13:58:34.365639 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 24629
I20250905 13:58:34.389081 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 24762
I20250905 13:58:34.414415 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 24895
I20250905 13:58:34.441675 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:41873
--webserver_interface=127.21.98.254
--webserver_port=42307
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:41873 with env {}
W20250905 13:58:34.730870 25165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:34.731530 25165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:34.731995 25165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:34.760968 25165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:58:34.761276 25165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:34.761546 25165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:58:34.761785 25165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:58:34.793646 25165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:41873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=42307
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:34.794849 25165 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:34.796418 25165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:34.805589 25171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:34.806864 25172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:36.053836 25174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:36.056669 25173 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1248 milliseconds
W20250905 13:58:36.058109 25165 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.252s user 0.391s sys 0.850s
W20250905 13:58:36.058354 25165 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.252s user 0.391s sys 0.850s
I20250905 13:58:36.058563 25165 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:36.059612 25165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:36.062170 25165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:36.063509 25165 hybrid_clock.cc:648] HybridClock initialized: now 1757080716063472 us; error 40 us; skew 500 ppm
I20250905 13:58:36.064263 25165 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:36.070950 25165 webserver.cc:480] Webserver started at http://127.21.98.254:42307/ using document root <none> and password file <none>
I20250905 13:58:36.071913 25165 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:36.072119 25165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:36.079797 25165 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20250905 13:58:36.084019 25181 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:36.085099 25165 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20250905 13:58:36.085415 25165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "705a9105a9124c8e93599291af08d21a"
format_stamp: "Formatted at 2025-09-05 13:58:22 on dist-test-slave-g001"
I20250905 13:58:36.087349 25165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:36.159976 25165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:36.161231 25165 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:36.161602 25165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:36.224164 25165 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:41873
I20250905 13:58:36.224236 25232 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:41873 every 8 connection(s)
I20250905 13:58:36.226974 25165 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 13:58:36.228772 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 25165
I20250905 13:58:36.230700 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:35763
--local_ip_for_outbound_sockets=127.21.98.193
--tserver_master_addrs=127.21.98.254:41873
--webserver_port=36177
--webserver_interface=127.21.98.193
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:58:36.242197 25233 sys_catalog.cc:263] Verifying existing consensus state
I20250905 13:58:36.247682 25233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Bootstrap starting.
I20250905 13:58:36.256935 25233 log.cc:826] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:36.300122 25233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Bootstrap replayed 1/1 log segments. Stats: ops{read=18 overwritten=0 applied=18 ignored=0} inserts{seen=13 ignored=0} mutations{seen=10 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:36.300823 25233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Bootstrap complete.
I20250905 13:58:36.320115 25233 raft_consensus.cc:357] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:36.321863 25233 raft_consensus.cc:738] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705a9105a9124c8e93599291af08d21a, State: Initialized, Role: FOLLOWER
I20250905 13:58:36.322474 25233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:36.322962 25233 raft_consensus.cc:397] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:36.323195 25233 raft_consensus.cc:491] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:36.323448 25233 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 2 FOLLOWER]: Advancing to term 3
I20250905 13:58:36.328579 25233 raft_consensus.cc:513] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:36.329121 25233 leader_election.cc:304] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 705a9105a9124c8e93599291af08d21a; no voters:
I20250905 13:58:36.331066 25233 leader_election.cc:290] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [CANDIDATE]: Term 3 election: Requested vote from peers
I20250905 13:58:36.331427 25237 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 3 FOLLOWER]: Leader election won for term 3
I20250905 13:58:36.334564 25237 raft_consensus.cc:695] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 3 LEADER]: Becoming Leader. State: Replica: 705a9105a9124c8e93599291af08d21a, State: Running, Role: LEADER
I20250905 13:58:36.335354 25237 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 18, Committed index: 18, Last appended: 2.18, Last appended by leader: 18, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:36.335908 25233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:58:36.342078 25239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 705a9105a9124c8e93599291af08d21a. Latest consensus state: current_term: 3 leader_uuid: "705a9105a9124c8e93599291af08d21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } } }
I20250905 13:58:36.343786 25239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: This master's current role is: LEADER
I20250905 13:58:36.343104 25238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "705a9105a9124c8e93599291af08d21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } } }
I20250905 13:58:36.345430 25238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: This master's current role is: LEADER
I20250905 13:58:36.355976 25244 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:58:36.367414 25244 catalog_manager.cc:671] Loaded metadata for table TestTable [id=4f86db6e8e6345638ccfccaeacb800ae]
I20250905 13:58:36.369097 25244 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=65b7003bcd99445888de06f6c59beb86]
I20250905 13:58:36.370640 25244 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=f8721b6cc75b452da9ca09f60e16b422]
I20250905 13:58:36.378058 25244 tablet_loader.cc:96] loaded metadata for tablet 2ef4f3179cbd4d71a473c89311a50045 (table TestTable [id=4f86db6e8e6345638ccfccaeacb800ae])
I20250905 13:58:36.379243 25244 tablet_loader.cc:96] loaded metadata for tablet 7c2039b8c6ed49eca474f06b27f83782 (table TestTable2 [id=f8721b6cc75b452da9ca09f60e16b422])
I20250905 13:58:36.380475 25244 tablet_loader.cc:96] loaded metadata for tablet b8ff2659b4504a0b827dbdd7041eaf50 (table TestTable1 [id=65b7003bcd99445888de06f6c59beb86])
I20250905 13:58:36.381788 25244 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:58:36.386754 25244 catalog_manager.cc:1261] Loaded cluster ID: 16f5536e4a9e4d689dd4678a1c856d3c
I20250905 13:58:36.387068 25244 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:58:36.394049 25244 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:58:36.399101 25244 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Loaded TSK: 0
I20250905 13:58:36.400499 25244 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 13:58:36.407431 25254 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250905 13:58:36.541075 25235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:36.541491 25235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:36.541936 25235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:36.569679 25235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:36.570374 25235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 13:58:36.600558 25235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:35763
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=36177
--tserver_master_addrs=127.21.98.254:41873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:36.601882 25235 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:36.603506 25235 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:36.615846 25260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:38.019296 25259 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 25235
W20250905 13:58:38.206596 25235 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.590s user 0.556s sys 0.991s
W20250905 13:58:38.206954 25235 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.590s user 0.556s sys 0.991s
W20250905 13:58:36.617194 25261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:38.208997 25263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:38.211377 25262 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1594 milliseconds
I20250905 13:58:38.211402 25235 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:38.212605 25235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:38.214545 25235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:38.215869 25235 hybrid_clock.cc:648] HybridClock initialized: now 1757080718215807 us; error 56 us; skew 500 ppm
I20250905 13:58:38.216641 25235 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:38.223500 25235 webserver.cc:480] Webserver started at http://127.21.98.193:36177/ using document root <none> and password file <none>
I20250905 13:58:38.224336 25235 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:38.224563 25235 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:38.232779 25235 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20250905 13:58:38.237481 25271 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:38.238457 25235 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20250905 13:58:38.238720 25235 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b41ea60da9974dabb2e4bb44807a7c9f"
format_stamp: "Formatted at 2025-09-05 13:58:24 on dist-test-slave-g001"
I20250905 13:58:38.240478 25235 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:38.293900 25235 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:38.295255 25235 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:38.295706 25235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:38.298884 25235 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:38.303969 25278 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 13:58:38.314481 25235 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 13:58:38.314671 25235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.000s sys 0.002s
I20250905 13:58:38.314909 25235 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 13:58:38.319164 25235 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 13:58:38.319320 25235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20250905 13:58:38.319713 25278 tablet_bootstrap.cc:492] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap starting.
I20250905 13:58:38.381111 25278 log.cc:826] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:38.474079 25278 tablet_bootstrap.cc:492] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:38.475174 25278 tablet_bootstrap.cc:492] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap complete.
I20250905 13:58:38.477034 25278 ts_tablet_manager.cc:1397] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent bootstrapping tablet: real 0.158s user 0.111s sys 0.042s
I20250905 13:58:38.497684 25278 raft_consensus.cc:357] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:38.500389 25278 raft_consensus.cc:738] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Initialized, Role: FOLLOWER
I20250905 13:58:38.501430 25278 consensus_queue.cc:260] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:38.502099 25278 raft_consensus.cc:397] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:38.502449 25278 raft_consensus.cc:491] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:38.502815 25278 raft_consensus.cc:3058] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 1 FOLLOWER]: Advancing to term 2
I20250905 13:58:38.510092 25278 raft_consensus.cc:513] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:38.510684 25235 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:35763
I20250905 13:58:38.510793 25385 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:35763 every 8 connection(s)
I20250905 13:58:38.510941 25278 leader_election.cc:304] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b41ea60da9974dabb2e4bb44807a7c9f; no voters:
I20250905 13:58:38.512748 25278 leader_election.cc:290] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 13:58:38.513161 25386 raft_consensus.cc:2802] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Leader election won for term 2
I20250905 13:58:38.513832 25235 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 13:58:38.521052 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 25235
I20250905 13:58:38.522192 25278 ts_tablet_manager.cc:1428] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent starting tablet: real 0.045s user 0.035s sys 0.007s
I20250905 13:58:38.523833 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.194:39317
--local_ip_for_outbound_sockets=127.21.98.194
--tserver_master_addrs=127.21.98.254:41873
--webserver_port=41835
--webserver_interface=127.21.98.194
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:58:38.523865 25386 raft_consensus.cc:695] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEADER]: Becoming Leader. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Running, Role: LEADER
I20250905 13:58:38.528731 25386 consensus_queue.cc:237] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:38.554906 25389 heartbeater.cc:344] Connected to a master server at 127.21.98.254:41873
I20250905 13:58:38.555366 25389 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:38.556531 25389 heartbeater.cc:507] Master 127.21.98.254:41873 requested a full tablet report, sending...
I20250905 13:58:38.560986 25198 ts_manager.cc:194] Registered new tserver with Master: b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193:35763)
I20250905 13:58:38.564774 25198 catalog_manager.cc:5582] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } health_report { overall_health: HEALTHY } } }
I20250905 13:58:38.608161 25198 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:44307
I20250905 13:58:38.612233 25389 heartbeater.cc:499] Master 127.21.98.254:41873 was elected leader, sending a full tablet report...
W20250905 13:58:38.831509 25395 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:38.831956 25395 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:38.832442 25395 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:38.859874 25395 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:38.860631 25395 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.194
I20250905 13:58:38.892985 25395 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.194:39317
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.98.194
--webserver_port=41835
--tserver_master_addrs=127.21.98.254:41873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.194
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:38.894238 25395 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:38.895805 25395 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:38.907562 25405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:38.909224 25406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:40.292121 25408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:40.294799 25407 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1380 milliseconds
W20250905 13:58:40.295648 25395 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.387s user 0.418s sys 0.949s
W20250905 13:58:40.295915 25395 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.388s user 0.418s sys 0.949s
I20250905 13:58:40.296139 25395 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:40.297190 25395 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:40.299801 25395 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:40.301195 25395 hybrid_clock.cc:648] HybridClock initialized: now 1757080720301138 us; error 59 us; skew 500 ppm
I20250905 13:58:40.301959 25395 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:40.310099 25395 webserver.cc:480] Webserver started at http://127.21.98.194:41835/ using document root <none> and password file <none>
I20250905 13:58:40.311103 25395 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:40.311347 25395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:40.319587 25395 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20250905 13:58:40.324779 25415 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:40.325827 25395 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20250905 13:58:40.326123 25395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c656231d41e2493f955684515def253f"
format_stamp: "Formatted at 2025-09-05 13:58:26 on dist-test-slave-g001"
I20250905 13:58:40.328131 25395 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:40.393654 25395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:40.395067 25395 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:40.395541 25395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:40.398087 25395 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:40.403117 25422 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 13:58:40.410197 25395 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 13:58:40.410526 25395 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.001s
I20250905 13:58:40.410768 25395 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 13:58:40.415237 25395 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 13:58:40.415477 25395 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20250905 13:58:40.415814 25422 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Bootstrap starting.
I20250905 13:58:40.469008 25422 log.cc:826] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:40.559376 25422 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:40.560361 25422 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Bootstrap complete.
I20250905 13:58:40.561610 25422 ts_tablet_manager.cc:1397] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Time spent bootstrapping tablet: real 0.146s user 0.099s sys 0.041s
I20250905 13:58:40.576539 25422 raft_consensus.cc:357] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } }
I20250905 13:58:40.578439 25422 raft_consensus.cc:738] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c656231d41e2493f955684515def253f, State: Initialized, Role: FOLLOWER
I20250905 13:58:40.579310 25422 consensus_queue.cc:260] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } }
I20250905 13:58:40.579777 25422 raft_consensus.cc:397] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:40.580065 25422 raft_consensus.cc:491] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:40.580322 25422 raft_consensus.cc:3058] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 1 FOLLOWER]: Advancing to term 2
I20250905 13:58:40.586218 25422 raft_consensus.cc:513] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } }
I20250905 13:58:40.587122 25422 leader_election.cc:304] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c656231d41e2493f955684515def253f; no voters:
I20250905 13:58:40.589196 25422 leader_election.cc:290] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 13:58:40.589555 25526 raft_consensus.cc:2802] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Leader election won for term 2
I20250905 13:58:40.592097 25526 raft_consensus.cc:695] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 LEADER]: Becoming Leader. State: Replica: c656231d41e2493f955684515def253f, State: Running, Role: LEADER
I20250905 13:58:40.593133 25526 consensus_queue.cc:237] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } }
I20250905 13:58:40.596916 25422 ts_tablet_manager.cc:1428] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Time spent starting tablet: real 0.035s user 0.031s sys 0.003s
I20250905 13:58:40.597648 25395 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.194:39317
I20250905 13:58:40.597963 25533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.194:39317 every 8 connection(s)
I20250905 13:58:40.600492 25395 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 13:58:40.605443 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 25395
I20250905 13:58:40.607502 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.195:43931
--local_ip_for_outbound_sockets=127.21.98.195
--tserver_master_addrs=127.21.98.254:41873
--webserver_port=36707
--webserver_interface=127.21.98.195
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:58:40.622771 25535 heartbeater.cc:344] Connected to a master server at 127.21.98.254:41873
I20250905 13:58:40.623155 25535 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:40.624100 25535 heartbeater.cc:507] Master 127.21.98.254:41873 requested a full tablet report, sending...
I20250905 13:58:40.627457 25197 ts_manager.cc:194] Registered new tserver with Master: c656231d41e2493f955684515def253f (127.21.98.194:39317)
I20250905 13:58:40.628564 25197 catalog_manager.cc:5582] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f reported cstate change: term changed from 0 to 2, leader changed from <none> to c656231d41e2493f955684515def253f (127.21.98.194), VOTER c656231d41e2493f955684515def253f (127.21.98.194) added. New cstate: current_term: 2 leader_uuid: "c656231d41e2493f955684515def253f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } health_report { overall_health: HEALTHY } } }
I20250905 13:58:40.641379 25197 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.194:45789
I20250905 13:58:40.645422 25535 heartbeater.cc:499] Master 127.21.98.254:41873 was elected leader, sending a full tablet report...
I20250905 13:58:40.656479 25485 consensus_queue.cc:237] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } }
I20250905 13:58:40.659643 25526 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } } }
I20250905 13:58:40.668370 25183 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2ef4f3179cbd4d71a473c89311a50045 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250905 13:58:40.671526 25197 catalog_manager.cc:5582] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f reported cstate change: config changed from index -1 to 9, NON_VOTER b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) added. New cstate: current_term: 2 leader_uuid: "c656231d41e2493f955684515def253f" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250905 13:58:40.673604 25419 consensus_peers.cc:489] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f -> Peer b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193:35763): Couldn't send request to peer b41ea60da9974dabb2e4bb44807a7c9f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ef4f3179cbd4d71a473c89311a50045. This is attempt 1: this message will repeat every 5th retry.
W20250905 13:58:40.679193 25197 catalog_manager.cc:5260] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2ef4f3179cbd4d71a473c89311a50045 with cas_config_opid_index 9: no extra replica candidate found for tablet 2ef4f3179cbd4d71a473c89311a50045 (table TestTable [id=4f86db6e8e6345638ccfccaeacb800ae]): Not found: could not select location for extra replica: not enough tablet servers to satisfy replica placement policy: the total number of registered tablet servers (2) does not allow for adding an extra replica; consider bringing up more to have at least 4 tablet servers up and running
W20250905 13:58:40.908140 25539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:40.908629 25539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:40.909121 25539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:40.938261 25539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:40.939074 25539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.195
I20250905 13:58:40.971866 25539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.195:43931
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.98.195
--webserver_port=36707
--tserver_master_addrs=127.21.98.254:41873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.195
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:40.973111 25539 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:40.974647 25539 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:40.985598 25552 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:58:41.147496 25558 ts_tablet_manager.cc:927] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Initiating tablet copy from peer c656231d41e2493f955684515def253f (127.21.98.194:39317)
I20250905 13:58:41.149856 25558 tablet_copy_client.cc:323] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: tablet copy: Beginning tablet copy session from remote peer at address 127.21.98.194:39317
I20250905 13:58:41.161433 25505 tablet_copy_service.cc:140] P c656231d41e2493f955684515def253f: Received BeginTabletCopySession request for tablet 2ef4f3179cbd4d71a473c89311a50045 from peer b41ea60da9974dabb2e4bb44807a7c9f ({username='slave'} at 127.21.98.193:37037)
I20250905 13:58:41.161825 25505 tablet_copy_service.cc:161] P c656231d41e2493f955684515def253f: Beginning new tablet copy session on tablet 2ef4f3179cbd4d71a473c89311a50045 from peer b41ea60da9974dabb2e4bb44807a7c9f at {username='slave'} at 127.21.98.193:37037: session id = b41ea60da9974dabb2e4bb44807a7c9f-2ef4f3179cbd4d71a473c89311a50045
I20250905 13:58:41.165910 25505 tablet_copy_source_session.cc:215] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 13:58:41.169071 25558 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef4f3179cbd4d71a473c89311a50045. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:41.179942 25558 tablet_copy_client.cc:806] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: tablet copy: Starting download of 0 data blocks...
I20250905 13:58:41.180469 25558 tablet_copy_client.cc:670] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: tablet copy: Starting download of 1 WAL segments...
I20250905 13:58:41.184489 25558 tablet_copy_client.cc:538] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 13:58:41.189700 25558 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap starting.
I20250905 13:58:41.284340 25558 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:41.284933 25558 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap complete.
I20250905 13:58:41.285353 25558 ts_tablet_manager.cc:1397] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent bootstrapping tablet: real 0.096s user 0.095s sys 0.000s
I20250905 13:58:41.287060 25558 raft_consensus.cc:357] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } }
I20250905 13:58:41.287490 25558 raft_consensus.cc:738] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Initialized, Role: LEARNER
I20250905 13:58:41.287904 25558 consensus_queue.cc:260] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } }
I20250905 13:58:41.290231 25558 ts_tablet_manager.cc:1428] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20250905 13:58:41.291852 25505 tablet_copy_service.cc:342] P c656231d41e2493f955684515def253f: Request end of tablet copy session b41ea60da9974dabb2e4bb44807a7c9f-2ef4f3179cbd4d71a473c89311a50045 received from {username='slave'} at 127.21.98.193:37037
I20250905 13:58:41.292198 25505 tablet_copy_service.cc:434] P c656231d41e2493f955684515def253f: ending tablet copy session b41ea60da9974dabb2e4bb44807a7c9f-2ef4f3179cbd4d71a473c89311a50045 on tablet 2ef4f3179cbd4d71a473c89311a50045 with peer b41ea60da9974dabb2e4bb44807a7c9f
I20250905 13:58:41.750530 25341 raft_consensus.cc:1215] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.8->[2.9-2.9] Dedup: 2.9->[]
W20250905 13:58:40.987174 25553 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:40.990360 25555 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:42.092931 25554 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20250905 13:58:42.093050 25539 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:42.096853 25539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:42.099494 25539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:42.100862 25539 hybrid_clock.cc:648] HybridClock initialized: now 1757080722100841 us; error 43 us; skew 500 ppm
I20250905 13:58:42.101538 25539 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:42.107403 25539 webserver.cc:480] Webserver started at http://127.21.98.195:36707/ using document root <none> and password file <none>
I20250905 13:58:42.108275 25539 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:42.108492 25539 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:42.115696 25539 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20250905 13:58:42.119889 25570 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:42.120760 25539 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20250905 13:58:42.121014 25539 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "867e2a35e49a4320b858ce1efad4e408"
format_stamp: "Formatted at 2025-09-05 13:58:28 on dist-test-slave-g001"
I20250905 13:58:42.122764 25539 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:42.162385 25563 raft_consensus.cc:1062] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: attempting to promote NON_VOTER b41ea60da9974dabb2e4bb44807a7c9f to VOTER
I20250905 13:58:42.163823 25563 consensus_queue.cc:237] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:42.167668 25341 raft_consensus.cc:1273] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEARNER]: Refusing update from remote peer c656231d41e2493f955684515def253f: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20250905 13:58:42.168787 25565 consensus_queue.cc:1035] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20250905 13:58:42.174527 25563 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } }
I20250905 13:58:42.176075 25341 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } }
I20250905 13:58:42.177258 25539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:42.179091 25539 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:42.179603 25539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:42.182654 25539 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:42.183048 25198 catalog_manager.cc:5582] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f reported cstate change: config changed from index 9 to 10, b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c656231d41e2493f955684515def253f" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 13:58:42.189991 25581 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 13:58:42.198591 25539 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 13:58:42.198834 25539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.002s sys 0.001s
I20250905 13:58:42.199170 25539 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 13:58:42.203538 25539 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 13:58:42.203697 25539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.004s sys 0.000s
I20250905 13:58:42.203985 25581 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap starting.
I20250905 13:58:42.260840 25581 log.cc:826] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:42.347767 25581 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:42.348515 25581 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap complete.
I20250905 13:58:42.349651 25581 ts_tablet_manager.cc:1397] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Time spent bootstrapping tablet: real 0.146s user 0.098s sys 0.043s
I20250905 13:58:42.366225 25581 raft_consensus.cc:357] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } }
I20250905 13:58:42.368079 25581 raft_consensus.cc:738] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867e2a35e49a4320b858ce1efad4e408, State: Initialized, Role: FOLLOWER
I20250905 13:58:42.369235 25581 consensus_queue.cc:260] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } }
I20250905 13:58:42.369726 25581 raft_consensus.cc:397] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:42.369962 25581 raft_consensus.cc:491] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:42.370208 25581 raft_consensus.cc:3058] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 1 FOLLOWER]: Advancing to term 2
I20250905 13:58:42.376446 25581 raft_consensus.cc:513] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } }
I20250905 13:58:42.377395 25581 leader_election.cc:304] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 867e2a35e49a4320b858ce1efad4e408; no voters:
I20250905 13:58:42.380102 25581 leader_election.cc:290] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 13:58:42.380409 25687 raft_consensus.cc:2802] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 13:58:42.384465 25687 raft_consensus.cc:695] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 2 LEADER]: Becoming Leader. State: Replica: 867e2a35e49a4320b858ce1efad4e408, State: Running, Role: LEADER
I20250905 13:58:42.385435 25687 consensus_queue.cc:237] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } }
I20250905 13:58:42.388310 25581 ts_tablet_manager.cc:1428] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Time spent starting tablet: real 0.038s user 0.025s sys 0.014s
I20250905 13:58:42.391081 25539 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.195:43931
I20250905 13:58:42.392299 25694 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.195:43931 every 8 connection(s)
I20250905 13:58:42.394030 25539 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 13:58:42.403331 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 25539
I20250905 13:58:42.410652 25696 heartbeater.cc:344] Connected to a master server at 127.21.98.254:41873
I20250905 13:58:42.411015 25696 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:42.411940 25696 heartbeater.cc:507] Master 127.21.98.254:41873 requested a full tablet report, sending...
I20250905 13:58:42.414983 25198 ts_manager.cc:194] Registered new tserver with Master: 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931)
I20250905 13:58:42.416162 25198 catalog_manager.cc:5582] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 reported cstate change: term changed from 0 to 2, leader changed from <none> to 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195), VOTER 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) added. New cstate: current_term: 2 leader_uuid: "867e2a35e49a4320b858ce1efad4e408" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } health_report { overall_health: HEALTHY } } }
I20250905 13:58:42.418349 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:58:42.422394 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:58:42.425300 25198 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.195:44351
W20250905 13:58:42.427280 21899 ts_itest-base.cc:209] found only 2 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER } interned_replicas { ts_info_idx: 1 role: FOLLOWER }
I20250905 13:58:42.429316 25696 heartbeater.cc:499] Master 127.21.98.254:41873 was elected leader, sending a full tablet report...
I20250905 13:58:42.438022 25646 consensus_queue.cc:237] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: NON_VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: true } }
I20250905 13:58:42.441166 25690 raft_consensus.cc:2953] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index -1 to 9, NON_VOTER c656231d41e2493f955684515def253f (127.21.98.194) added. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: NON_VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: true } } }
I20250905 13:58:42.449492 25183 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b8ff2659b4504a0b827dbdd7041eaf50 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20250905 13:58:42.451033 25572 consensus_peers.cc:489] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 -> Peer c656231d41e2493f955684515def253f (127.21.98.194:39317): Couldn't send request to peer c656231d41e2493f955684515def253f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8ff2659b4504a0b827dbdd7041eaf50. This is attempt 1: this message will repeat every 5th retry.
I20250905 13:58:42.451579 25198 catalog_manager.cc:5582] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 reported cstate change: config changed from index -1 to 9, NON_VOTER c656231d41e2493f955684515def253f (127.21.98.194) added. New cstate: current_term: 2 leader_uuid: "867e2a35e49a4320b858ce1efad4e408" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: NON_VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 13:58:42.460098 25646 consensus_queue.cc:237] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: NON_VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } }
I20250905 13:58:42.463011 25690 raft_consensus.cc:2953] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, NON_VOTER b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) added. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: NON_VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } } }
W20250905 13:58:42.464792 25572 consensus_peers.cc:489] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 -> Peer c656231d41e2493f955684515def253f (127.21.98.194:39317): Couldn't send request to peer c656231d41e2493f955684515def253f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8ff2659b4504a0b827dbdd7041eaf50. This is attempt 1: this message will repeat every 5th retry.
I20250905 13:58:42.469311 25183 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b8ff2659b4504a0b827dbdd7041eaf50 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20250905 13:58:42.472280 25198 catalog_manager.cc:5582] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 reported cstate change: config changed from index 9 to 10, NON_VOTER b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) added. New cstate: current_term: 2 leader_uuid: "867e2a35e49a4320b858ce1efad4e408" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: NON_VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20250905 13:58:42.473457 25574 consensus_peers.cc:489] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 -> Peer b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193:35763): Couldn't send request to peer b41ea60da9974dabb2e4bb44807a7c9f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b8ff2659b4504a0b827dbdd7041eaf50. This is attempt 1: this message will repeat every 5th retry.
I20250905 13:58:42.539927 25485 consensus_queue.cc:237] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: NON_VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: true } }
I20250905 13:58:42.543850 25341 raft_consensus.cc:1273] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Refusing update from remote peer c656231d41e2493f955684515def253f: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250905 13:58:42.545028 25563 consensus_queue.cc:1035] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250905 13:58:42.550519 25565 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: NON_VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: true } } }
I20250905 13:58:42.552078 25341 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) added. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: NON_VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: true } } }
W20250905 13:58:42.553757 25417 consensus_peers.cc:489] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f -> Peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): Couldn't send request to peer 867e2a35e49a4320b858ce1efad4e408. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2ef4f3179cbd4d71a473c89311a50045. This is attempt 1: this message will repeat every 5th retry.
I20250905 13:58:42.557852 25183 catalog_manager.cc:5095] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2ef4f3179cbd4d71a473c89311a50045 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 5)
I20250905 13:58:42.560748 25198 catalog_manager.cc:5582] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f reported cstate change: config changed from index 10 to 11, NON_VOTER 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) added. New cstate: current_term: 2 leader_uuid: "c656231d41e2493f955684515def253f" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: NON_VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 13:58:42.845456 25707 ts_tablet_manager.cc:927] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Initiating tablet copy from peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931)
I20250905 13:58:42.847483 25707 tablet_copy_client.cc:323] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: tablet copy: Beginning tablet copy session from remote peer at address 127.21.98.195:43931
I20250905 13:58:42.848995 25666 tablet_copy_service.cc:140] P 867e2a35e49a4320b858ce1efad4e408: Received BeginTabletCopySession request for tablet b8ff2659b4504a0b827dbdd7041eaf50 from peer c656231d41e2493f955684515def253f ({username='slave'} at 127.21.98.194:32863)
I20250905 13:58:42.849417 25666 tablet_copy_service.cc:161] P 867e2a35e49a4320b858ce1efad4e408: Beginning new tablet copy session on tablet b8ff2659b4504a0b827dbdd7041eaf50 from peer c656231d41e2493f955684515def253f at {username='slave'} at 127.21.98.194:32863: session id = c656231d41e2493f955684515def253f-b8ff2659b4504a0b827dbdd7041eaf50
I20250905 13:58:42.853416 25666 tablet_copy_source_session.cc:215] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 13:58:42.856408 25707 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ff2659b4504a0b827dbdd7041eaf50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:42.865984 25707 tablet_copy_client.cc:806] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: tablet copy: Starting download of 0 data blocks...
I20250905 13:58:42.866387 25707 tablet_copy_client.cc:670] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: tablet copy: Starting download of 1 WAL segments...
I20250905 13:58:42.869576 25707 tablet_copy_client.cc:538] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 13:58:42.874228 25707 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Bootstrap starting.
I20250905 13:58:42.951426 25707 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:42.951970 25707 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Bootstrap complete.
I20250905 13:58:42.952358 25707 ts_tablet_manager.cc:1397] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Time spent bootstrapping tablet: real 0.078s user 0.072s sys 0.008s
I20250905 13:58:42.953965 25707 raft_consensus.cc:357] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: NON_VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } }
I20250905 13:58:42.954389 25707 raft_consensus.cc:738] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c656231d41e2493f955684515def253f, State: Initialized, Role: LEARNER
I20250905 13:58:42.954809 25707 consensus_queue.cc:260] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: NON_VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } }
I20250905 13:58:42.956210 25707 ts_tablet_manager.cc:1428] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250905 13:58:42.957743 25666 tablet_copy_service.cc:342] P 867e2a35e49a4320b858ce1efad4e408: Request end of tablet copy session c656231d41e2493f955684515def253f-b8ff2659b4504a0b827dbdd7041eaf50 received from {username='slave'} at 127.21.98.194:32863
I20250905 13:58:42.958194 25666 tablet_copy_service.cc:434] P 867e2a35e49a4320b858ce1efad4e408: ending tablet copy session c656231d41e2493f955684515def253f-b8ff2659b4504a0b827dbdd7041eaf50 on tablet b8ff2659b4504a0b827dbdd7041eaf50 with peer c656231d41e2493f955684515def253f
I20250905 13:58:42.976462 25183 catalog_manager.cc:5129] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2ef4f3179cbd4d71a473c89311a50045 with cas_config_opid_index 9: aborting the task: latest config opid_index 11; task opid_index 9
I20250905 13:58:42.992058 25710 ts_tablet_manager.cc:927] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Initiating tablet copy from peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931)
I20250905 13:58:42.993140 25710 tablet_copy_client.cc:323] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: tablet copy: Beginning tablet copy session from remote peer at address 127.21.98.195:43931
I20250905 13:58:42.998040 25713 ts_tablet_manager.cc:927] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Initiating tablet copy from peer c656231d41e2493f955684515def253f (127.21.98.194:39317)
I20250905 13:58:43.000089 25713 tablet_copy_client.cc:323] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: tablet copy: Beginning tablet copy session from remote peer at address 127.21.98.194:39317
I20250905 13:58:43.001896 25505 tablet_copy_service.cc:140] P c656231d41e2493f955684515def253f: Received BeginTabletCopySession request for tablet 2ef4f3179cbd4d71a473c89311a50045 from peer 867e2a35e49a4320b858ce1efad4e408 ({username='slave'} at 127.21.98.195:55597)
I20250905 13:58:43.002283 25505 tablet_copy_service.cc:161] P c656231d41e2493f955684515def253f: Beginning new tablet copy session on tablet 2ef4f3179cbd4d71a473c89311a50045 from peer 867e2a35e49a4320b858ce1efad4e408 at {username='slave'} at 127.21.98.195:55597: session id = 867e2a35e49a4320b858ce1efad4e408-2ef4f3179cbd4d71a473c89311a50045
I20250905 13:58:43.002569 25666 tablet_copy_service.cc:140] P 867e2a35e49a4320b858ce1efad4e408: Received BeginTabletCopySession request for tablet b8ff2659b4504a0b827dbdd7041eaf50 from peer b41ea60da9974dabb2e4bb44807a7c9f ({username='slave'} at 127.21.98.193:39435)
I20250905 13:58:43.003005 25666 tablet_copy_service.cc:161] P 867e2a35e49a4320b858ce1efad4e408: Beginning new tablet copy session on tablet b8ff2659b4504a0b827dbdd7041eaf50 from peer b41ea60da9974dabb2e4bb44807a7c9f at {username='slave'} at 127.21.98.193:39435: session id = b41ea60da9974dabb2e4bb44807a7c9f-b8ff2659b4504a0b827dbdd7041eaf50
I20250905 13:58:43.008848 25505 tablet_copy_source_session.cc:215] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 13:58:43.009275 25666 tablet_copy_source_session.cc:215] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Tablet Copy: opened 0 blocks and 1 log segments
I20250905 13:58:43.011477 25710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8ff2659b4504a0b827dbdd7041eaf50. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:43.011543 25713 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ef4f3179cbd4d71a473c89311a50045. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:58:43.020704 25710 tablet_copy_client.cc:806] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: tablet copy: Starting download of 0 data blocks...
I20250905 13:58:43.021104 25710 tablet_copy_client.cc:670] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: tablet copy: Starting download of 1 WAL segments...
I20250905 13:58:43.022902 25713 tablet_copy_client.cc:806] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: tablet copy: Starting download of 0 data blocks...
I20250905 13:58:43.023377 25713 tablet_copy_client.cc:670] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: tablet copy: Starting download of 1 WAL segments...
I20250905 13:58:43.024328 25710 tablet_copy_client.cc:538] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 13:58:43.026724 25713 tablet_copy_client.cc:538] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20250905 13:58:43.030697 25710 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap starting.
I20250905 13:58:43.034332 25713 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap starting.
I20250905 13:58:43.135555 25710 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:43.136171 25710 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap complete.
I20250905 13:58:43.136618 25710 ts_tablet_manager.cc:1397] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent bootstrapping tablet: real 0.106s user 0.096s sys 0.012s
I20250905 13:58:43.138245 25710 raft_consensus.cc:357] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: NON_VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } }
I20250905 13:58:43.138708 25710 raft_consensus.cc:738] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Initialized, Role: LEARNER
I20250905 13:58:43.138927 25713 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:43.139139 25710 consensus_queue.cc:260] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 2.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: NON_VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: true } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } }
I20250905 13:58:43.139465 25713 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap complete.
I20250905 13:58:43.139864 25713 ts_tablet_manager.cc:1397] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Time spent bootstrapping tablet: real 0.106s user 0.104s sys 0.004s
I20250905 13:58:43.141091 25710 ts_tablet_manager.cc:1428] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent starting tablet: real 0.004s user 0.001s sys 0.004s
I20250905 13:58:43.141649 25713 raft_consensus.cc:357] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: NON_VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: true } }
I20250905 13:58:43.142246 25713 raft_consensus.cc:738] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 867e2a35e49a4320b858ce1efad4e408, State: Initialized, Role: LEARNER
I20250905 13:58:43.142683 25666 tablet_copy_service.cc:342] P 867e2a35e49a4320b858ce1efad4e408: Request end of tablet copy session b41ea60da9974dabb2e4bb44807a7c9f-b8ff2659b4504a0b827dbdd7041eaf50 received from {username='slave'} at 127.21.98.193:39435
I20250905 13:58:43.143020 25666 tablet_copy_service.cc:434] P 867e2a35e49a4320b858ce1efad4e408: ending tablet copy session b41ea60da9974dabb2e4bb44807a7c9f-b8ff2659b4504a0b827dbdd7041eaf50 on tablet b8ff2659b4504a0b827dbdd7041eaf50 with peer b41ea60da9974dabb2e4bb44807a7c9f
I20250905 13:58:43.142825 25713 consensus_queue.cc:260] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: NON_VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: true } }
I20250905 13:58:43.144661 25713 ts_tablet_manager.cc:1428] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20250905 13:58:43.146103 25505 tablet_copy_service.cc:342] P c656231d41e2493f955684515def253f: Request end of tablet copy session 867e2a35e49a4320b858ce1efad4e408-2ef4f3179cbd4d71a473c89311a50045 received from {username='slave'} at 127.21.98.195:55597
I20250905 13:58:43.146499 25505 tablet_copy_service.cc:434] P c656231d41e2493f955684515def253f: ending tablet copy session 867e2a35e49a4320b858ce1efad4e408-2ef4f3179cbd4d71a473c89311a50045 on tablet 2ef4f3179cbd4d71a473c89311a50045 with peer 867e2a35e49a4320b858ce1efad4e408
I20250905 13:58:43.245378 25485 raft_consensus.cc:1215] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250905 13:58:43.431397 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b41ea60da9974dabb2e4bb44807a7c9f to finish bootstrapping
I20250905 13:58:43.446029 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver c656231d41e2493f955684515def253f to finish bootstrapping
I20250905 13:58:43.456291 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 867e2a35e49a4320b858ce1efad4e408 to finish bootstrapping
I20250905 13:58:43.532440 25646 raft_consensus.cc:1215] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
I20250905 13:58:43.607566 25341 raft_consensus.cc:1215] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEARNER]: Deduplicated request from leader. Original: 2.9->[2.10-2.10] Dedup: 2.10->[]
I20250905 13:58:43.665495 25687 raft_consensus.cc:1062] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: attempting to promote NON_VOTER c656231d41e2493f955684515def253f to VOTER
I20250905 13:58:43.667248 25687 consensus_queue.cc:237] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } }
I20250905 13:58:43.672500 25485 raft_consensus.cc:1273] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 LEARNER]: Refusing update from remote peer 867e2a35e49a4320b858ce1efad4e408: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250905 13:58:43.672685 25341 raft_consensus.cc:1273] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEARNER]: Refusing update from remote peer 867e2a35e49a4320b858ce1efad4e408: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
I20250905 13:58:43.673722 25687 consensus_queue.cc:1035] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.001s
I20250905 13:58:43.674649 25719 consensus_queue.cc:1035] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20250905 13:58:43.682684 25718 raft_consensus.cc:2953] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, c656231d41e2493f955684515def253f (127.21.98.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } } }
I20250905 13:58:43.684511 25484 raft_consensus.cc:2953] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, c656231d41e2493f955684515def253f (127.21.98.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } } }
I20250905 13:58:43.697353 25341 raft_consensus.cc:2953] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEARNER]: Committing config change with OpId 2.11: config changed from index 10 to 11, c656231d41e2493f955684515def253f (127.21.98.194) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } } }
I20250905 13:58:43.697624 25197 catalog_manager.cc:5582] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 reported cstate change: config changed from index 10 to 11, c656231d41e2493f955684515def253f (127.21.98.194) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "867e2a35e49a4320b858ce1efad4e408" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: NON_VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20250905 13:58:43.702208 25687 raft_consensus.cc:1062] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: attempting to promote NON_VOTER b41ea60da9974dabb2e4bb44807a7c9f to VOTER
I20250905 13:58:43.704352 25687 consensus_queue.cc:237] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:43.713646 25484 raft_consensus.cc:1273] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Refusing update from remote peer 867e2a35e49a4320b858ce1efad4e408: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250905 13:58:43.714460 25341 raft_consensus.cc:1273] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 LEARNER]: Refusing update from remote peer 867e2a35e49a4320b858ce1efad4e408: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250905 13:58:43.715256 25719 consensus_queue.cc:1035] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20250905 13:58:43.717211 25718 consensus_queue.cc:1035] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250905 13:58:43.728586 25719 raft_consensus.cc:2953] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } }
I20250905 13:58:43.730935 25341 raft_consensus.cc:2953] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } }
I20250905 13:58:43.737107 25484 raft_consensus.cc:2953] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } }
I20250905 13:58:43.752151 25194 catalog_manager.cc:5582] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 reported cstate change: config changed from index 11 to 12, b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "867e2a35e49a4320b858ce1efad4e408" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 13:58:43.782745 25626 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 13:58:43.787742 25321 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 13:58:43.796772 25465 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
705a9105a9124c8e93599291af08d21a | 127.21.98.254:41873 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.98.212:43167 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
867e2a35e49a4320b858ce1efad4e408 | 127.21.98.195:43931 | HEALTHY | <none> | 1 | 0
b41ea60da9974dabb2e4bb44807a7c9f | 127.21.98.193:35763 | HEALTHY | <none> | 1 | 0
c656231d41e2493f955684515def253f | 127.21.98.194:39317 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.21.98.193 | experimental | 127.21.98.193:35763
local_ip_for_outbound_sockets | 127.21.98.194 | experimental | 127.21.98.194:39317
local_ip_for_outbound_sockets | 127.21.98.195 | experimental | 127.21.98.195:43931
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.21.98.193:35763
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.21.98.194:39317
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.21.98.195:43931
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.98.212:43167 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
Tablet b8ff2659b4504a0b827dbdd7041eaf50 of table 'TestTable1' is conflicted: 3 replicas' active configs disagree with the leader master's
867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): RUNNING [LEADER]
c656231d41e2493f955684515def253f (127.21.98.194:39317): RUNNING
b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193:35763): RUNNING [NONVOTER]
All reported replicas are:
A = 867e2a35e49a4320b858ce1efad4e408
B = c656231d41e2493f955684515def253f
C = b41ea60da9974dabb2e4bb44807a7c9f
The consensus matrix is:
Config source | Replicas | Current term | Config index | Committed?
---------------+--------------+--------------+--------------+------------
master | A* B C~ | | | Yes
A | A* B C | 2 | 12 | Yes
B | A* B C | 2 | 12 | Yes
C | A* B C | 2 | 12 | Yes
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+--------------------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | CONSENSUS_MISMATCH | 1 | 0 | 0 | 0 | 1
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
==================
Errors:
==================
Corruption: table consistency check error: 1 out of 3 table(s) are not healthy
FAILED
I20250905 13:58:44.021723 21899 cluster_verifier.cc:83] Check not successful yet, sleeping and retrying: Runtime error: ksck discovered errors
I20250905 13:58:44.073772 25731 raft_consensus.cc:1062] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: attempting to promote NON_VOTER 867e2a35e49a4320b858ce1efad4e408 to VOTER
I20250905 13:58:44.075234 25731 consensus_queue.cc:237] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:44.079656 25646 raft_consensus.cc:1273] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [term 2 LEARNER]: Refusing update from remote peer c656231d41e2493f955684515def253f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250905 13:58:44.080034 25341 raft_consensus.cc:1273] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Refusing update from remote peer c656231d41e2493f955684515def253f: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20250905 13:58:44.080664 25731 consensus_queue.cc:1035] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Connected to new peer: Peer: permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250905 13:58:44.081415 25757 consensus_queue.cc:1035] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20250905 13:58:44.087466 25731 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } } }
I20250905 13:58:44.088902 25341 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } } }
I20250905 13:58:44.089815 25646 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } } }
I20250905 13:58:44.096621 25197 catalog_manager.cc:5582] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f reported cstate change: config changed from index 11 to 12, 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c656231d41e2493f955684515def253f" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20250905 13:58:44.402268 25626 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 13:58:44.408054 25465 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 13:58:44.409790 25321 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
705a9105a9124c8e93599291af08d21a | 127.21.98.254:41873 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.98.212:43167 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
867e2a35e49a4320b858ce1efad4e408 | 127.21.98.195:43931 | HEALTHY | <none> | 1 | 0
b41ea60da9974dabb2e4bb44807a7c9f | 127.21.98.193:35763 | HEALTHY | <none> | 1 | 0
c656231d41e2493f955684515def253f | 127.21.98.194:39317 | HEALTHY | <none> | 1 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.21.98.193 | experimental | 127.21.98.193:35763
local_ip_for_outbound_sockets | 127.21.98.194 | experimental | 127.21.98.194:39317
local_ip_for_outbound_sockets | 127.21.98.195 | experimental | 127.21.98.195:43931
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.21.98.193:35763
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.21.98.194:39317
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.21.98.195:43931
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.98.212:43167 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable1 | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
TestTable2 | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 3
Tablets | 3
Replicas | 7
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250905 13:58:44.628018 21899 log_verifier.cc:126] Checking tablet 2ef4f3179cbd4d71a473c89311a50045
I20250905 13:58:44.708693 21899 log_verifier.cc:177] Verified matching terms for 12 ops in tablet 2ef4f3179cbd4d71a473c89311a50045
I20250905 13:58:44.709002 21899 log_verifier.cc:126] Checking tablet 7c2039b8c6ed49eca474f06b27f83782
I20250905 13:58:44.733623 21899 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 7c2039b8c6ed49eca474f06b27f83782
I20250905 13:58:44.733830 21899 log_verifier.cc:126] Checking tablet b8ff2659b4504a0b827dbdd7041eaf50
I20250905 13:58:44.810231 21899 log_verifier.cc:177] Verified matching terms for 12 ops in tablet b8ff2659b4504a0b827dbdd7041eaf50
I20250905 13:58:44.810683 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 25165
I20250905 13:58:44.834285 21899 minidump.cc:252] Setting minidump size limit to 20M
I20250905 13:58:44.835467 21899 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:44.836411 21899 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:44.845610 25793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:44.845791 25794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:44.847178 25796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:58:44.940943 21899 server_base.cc:1047] running on GCE node
I20250905 13:58:44.941956 21899 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20250905 13:58:44.942138 21899 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20250905 13:58:44.942260 21899 hybrid_clock.cc:648] HybridClock initialized: now 1757080724942245 us; error 0 us; skew 500 ppm
I20250905 13:58:44.942806 21899 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:44.945248 21899 webserver.cc:480] Webserver started at http://0.0.0.0:38189/ using document root <none> and password file <none>
I20250905 13:58:44.945936 21899 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:44.946081 21899 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:44.950573 21899 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20250905 13:58:44.953539 25801 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:44.954496 21899 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20250905 13:58:44.954757 21899 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "705a9105a9124c8e93599291af08d21a"
format_stamp: "Formatted at 2025-09-05 13:58:22 on dist-test-slave-g001"
I20250905 13:58:44.956306 21899 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:44.974685 21899 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:44.976372 21899 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:44.976795 21899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:44.984519 21899 sys_catalog.cc:263] Verifying existing consensus state
W20250905 13:58:44.988418 21899 sys_catalog.cc:243] For a single master config, on-disk Raft master: 127.21.98.254:41873 exists but no master address supplied!
I20250905 13:58:44.989987 21899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Bootstrap starting.
I20250905 13:58:45.028771 21899 log.cc:826] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:45.088094 21899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Bootstrap replayed 1/1 log segments. Stats: ops{read=30 overwritten=0 applied=30 ignored=0} inserts{seen=13 ignored=0} mutations{seen=21 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:45.088778 21899 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Bootstrap complete.
I20250905 13:58:45.101881 21899 raft_consensus.cc:357] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:45.102378 21899 raft_consensus.cc:738] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705a9105a9124c8e93599291af08d21a, State: Initialized, Role: FOLLOWER
I20250905 13:58:45.103044 21899 consensus_queue.cc:260] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:45.103471 21899 raft_consensus.cc:397] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 3 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:45.103685 21899 raft_consensus.cc:491] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 3 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:45.103935 21899 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 3 FOLLOWER]: Advancing to term 4
W20250905 13:58:45.109972 25696 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:41873: connect: Connection refused (error 111)
I20250905 13:58:45.109804 21899 raft_consensus.cc:513] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:45.110563 21899 leader_election.cc:304] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 705a9105a9124c8e93599291af08d21a; no voters:
W20250905 13:58:45.111053 25389 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:41873: connect: Connection refused (error 111)
I20250905 13:58:45.111948 21899 leader_election.cc:290] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [CANDIDATE]: Term 4 election: Requested vote from peers
I20250905 13:58:45.112205 25811 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 4 FOLLOWER]: Leader election won for term 4
W20250905 13:58:45.113095 25535 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:41873 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:41873: connect: Connection refused (error 111)
I20250905 13:58:45.113845 25811 raft_consensus.cc:695] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 4 LEADER]: Becoming Leader. State: Replica: 705a9105a9124c8e93599291af08d21a, State: Running, Role: LEADER
I20250905 13:58:45.114522 25811 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 30, Committed index: 30, Last appended: 3.30, Last appended by leader: 30, Current term: 4, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:45.121666 25812 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 4 leader_uuid: "705a9105a9124c8e93599291af08d21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } } }
I20250905 13:58:45.122187 25812 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: This master's current role is: LEADER
I20250905 13:58:45.122020 25813 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 705a9105a9124c8e93599291af08d21a. Latest consensus state: current_term: 4 leader_uuid: "705a9105a9124c8e93599291af08d21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } } }
I20250905 13:58:45.122538 25813 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: This master's current role is: LEADER
I20250905 13:58:45.145831 21899 tablet_replica.cc:331] stopping tablet replica
I20250905 13:58:45.146272 21899 raft_consensus.cc:2241] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 4 LEADER]: Raft consensus shutting down.
I20250905 13:58:45.146613 21899 raft_consensus.cc:2270] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 4 FOLLOWER]: Raft consensus is shut down!
I20250905 13:58:45.148655 21899 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250905 13:58:45.149089 21899 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250905 13:58:45.177019 21899 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250905 13:58:50.217442 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 25235
I20250905 13:58:50.245175 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 25395
I20250905 13:58:50.272547 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 25539
I20250905 13:58:50.303357 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:41873
--webserver_interface=127.21.98.254
--webserver_port=42307
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:41873 with env {}
W20250905 13:58:50.600698 25884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:50.601235 25884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:50.601656 25884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:50.631506 25884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:58:50.631829 25884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:50.632076 25884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:58:50.632302 25884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:58:50.665876 25884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:41873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:41873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=42307
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:50.667212 25884 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:50.668851 25884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:50.679622 25890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:50.680270 25891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:51.850965 25892 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1170 milliseconds
W20250905 13:58:51.852167 25893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:51.854519 25884 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.175s user 0.000s sys 0.006s
W20250905 13:58:51.854965 25884 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.176s user 0.001s sys 0.006s
I20250905 13:58:51.855250 25884 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:51.856889 25884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:51.860013 25884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:51.861526 25884 hybrid_clock.cc:648] HybridClock initialized: now 1757080731861482 us; error 30 us; skew 500 ppm
I20250905 13:58:51.862739 25884 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:51.871608 25884 webserver.cc:480] Webserver started at http://127.21.98.254:42307/ using document root <none> and password file <none>
I20250905 13:58:51.873015 25884 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:51.873291 25884 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:51.884398 25884 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.004s
I20250905 13:58:51.890746 25900 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:51.891954 25884 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20250905 13:58:51.892400 25884 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "705a9105a9124c8e93599291af08d21a"
format_stamp: "Formatted at 2025-09-05 13:58:22 on dist-test-slave-g001"
I20250905 13:58:51.894984 25884 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:51.968643 25884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:51.969973 25884 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:51.970332 25884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:52.038087 25884 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:41873
I20250905 13:58:52.038133 25951 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:41873 every 8 connection(s)
I20250905 13:58:52.040848 25884 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 13:58:52.043254 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 25884
I20250905 13:58:52.045306 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:35763
--local_ip_for_outbound_sockets=127.21.98.193
--tserver_master_addrs=127.21.98.254:41873
--webserver_port=36177
--webserver_interface=127.21.98.193
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:58:52.057740 25952 sys_catalog.cc:263] Verifying existing consensus state
I20250905 13:58:52.062217 25952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Bootstrap starting.
I20250905 13:58:52.071364 25952 log.cc:826] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:52.144433 25952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Bootstrap replayed 1/1 log segments. Stats: ops{read=34 overwritten=0 applied=34 ignored=0} inserts{seen=15 ignored=0} mutations{seen=23 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:52.145264 25952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Bootstrap complete.
I20250905 13:58:52.162689 25952 raft_consensus.cc:357] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 5 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:52.164548 25952 raft_consensus.cc:738] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 5 FOLLOWER]: Becoming Follower/Learner. State: Replica: 705a9105a9124c8e93599291af08d21a, State: Initialized, Role: FOLLOWER
I20250905 13:58:52.165277 25952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:52.165745 25952 raft_consensus.cc:397] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 5 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:52.166004 25952 raft_consensus.cc:491] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 5 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:52.166275 25952 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 5 FOLLOWER]: Advancing to term 6
I20250905 13:58:52.171453 25952 raft_consensus.cc:513] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:52.172084 25952 leader_election.cc:304] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [CANDIDATE]: Term 6 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 705a9105a9124c8e93599291af08d21a; no voters:
I20250905 13:58:52.174185 25952 leader_election.cc:290] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [CANDIDATE]: Term 6 election: Requested vote from peers
I20250905 13:58:52.174445 25956 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 6 FOLLOWER]: Leader election won for term 6
I20250905 13:58:52.177685 25956 raft_consensus.cc:695] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [term 6 LEADER]: Becoming Leader. State: Replica: 705a9105a9124c8e93599291af08d21a, State: Running, Role: LEADER
I20250905 13:58:52.178503 25956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 34, Committed index: 34, Last appended: 5.34, Last appended by leader: 34, Current term: 6, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } }
I20250905 13:58:52.179203 25952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:58:52.185693 25958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 705a9105a9124c8e93599291af08d21a. Latest consensus state: current_term: 6 leader_uuid: "705a9105a9124c8e93599291af08d21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } } }
I20250905 13:58:52.186419 25958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: This master's current role is: LEADER
I20250905 13:58:52.189029 25957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 6 leader_uuid: "705a9105a9124c8e93599291af08d21a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "705a9105a9124c8e93599291af08d21a" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 41873 } } }
I20250905 13:58:52.189730 25957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a [sys.catalog]: This master's current role is: LEADER
I20250905 13:58:52.202059 25962 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:58:52.215588 25962 catalog_manager.cc:671] Loaded metadata for table TestTable1 [id=65b7003bcd99445888de06f6c59beb86]
I20250905 13:58:52.217236 25962 catalog_manager.cc:671] Loaded metadata for table TestTable [id=af31f66eacc648f1925e1ef1ec580f02]
I20250905 13:58:52.218931 25962 catalog_manager.cc:671] Loaded metadata for table TestTable2 [id=f8721b6cc75b452da9ca09f60e16b422]
I20250905 13:58:52.227407 25962 tablet_loader.cc:96] loaded metadata for tablet 2ef4f3179cbd4d71a473c89311a50045 (table TestTable [id=af31f66eacc648f1925e1ef1ec580f02])
I20250905 13:58:52.228686 25962 tablet_loader.cc:96] loaded metadata for tablet 7c2039b8c6ed49eca474f06b27f83782 (table TestTable2 [id=f8721b6cc75b452da9ca09f60e16b422])
I20250905 13:58:52.230050 25962 tablet_loader.cc:96] loaded metadata for tablet b8ff2659b4504a0b827dbdd7041eaf50 (table TestTable1 [id=65b7003bcd99445888de06f6c59beb86])
I20250905 13:58:52.231240 25962 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:58:52.236378 25962 catalog_manager.cc:1261] Loaded cluster ID: 16f5536e4a9e4d689dd4678a1c856d3c
I20250905 13:58:52.236703 25962 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:58:52.244221 25962 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:58:52.249383 25962 catalog_manager.cc:5966] T 00000000000000000000000000000000 P 705a9105a9124c8e93599291af08d21a: Loaded TSK: 0
I20250905 13:58:52.250830 25962 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 13:58:52.254999 25974 catalog_manager.cc:797] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20250905 13:58:52.370661 25954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:52.371196 25954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:52.371742 25954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:52.399726 25954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:52.400485 25954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 13:58:52.432101 25954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:35763
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=36177
--tserver_master_addrs=127.21.98.254:41873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:52.433318 25954 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:52.434718 25954 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:52.446599 25980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:52.447713 25981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:52.451455 25983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:58:52.451426 25954 server_base.cc:1047] running on GCE node
I20250905 13:58:53.546331 25954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:53.549017 25954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:53.550555 25954 hybrid_clock.cc:648] HybridClock initialized: now 1757080733550504 us; error 44 us; skew 500 ppm
I20250905 13:58:53.551590 25954 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:53.559154 25954 webserver.cc:480] Webserver started at http://127.21.98.193:36177/ using document root <none> and password file <none>
I20250905 13:58:53.560328 25954 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:53.560649 25954 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:53.570603 25954 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20250905 13:58:53.576876 25990 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:53.577972 25954 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20250905 13:58:53.578377 25954 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "b41ea60da9974dabb2e4bb44807a7c9f"
format_stamp: "Formatted at 2025-09-05 13:58:24 on dist-test-slave-g001"
I20250905 13:58:53.581029 25954 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:53.623682 25954 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:53.624909 25954 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:53.625270 25954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:53.627578 25954 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:53.633361 25997 ts_tablet_manager.cc:542] Loading tablet metadata (0/3 complete)
I20250905 13:58:53.655850 25954 ts_tablet_manager.cc:579] Loaded tablet metadata (3 total tablets, 3 live tablets)
I20250905 13:58:53.656098 25954 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.024s user 0.001s sys 0.001s
I20250905 13:58:53.656366 25954 ts_tablet_manager.cc:594] Registering tablets (0/3 complete)
I20250905 13:58:53.661154 25997 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap starting.
I20250905 13:58:53.666196 25954 ts_tablet_manager.cc:610] Registered 3 tablets
I20250905 13:58:53.666365 25954 ts_tablet_manager.cc:589] Time spent register tablets: real 0.010s user 0.007s sys 0.000s
I20250905 13:58:53.726562 25997 log.cc:826] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:53.837916 25997 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:53.839036 25997 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap complete.
I20250905 13:58:53.840848 25997 ts_tablet_manager.cc:1397] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent bootstrapping tablet: real 0.180s user 0.152s sys 0.024s
I20250905 13:58:53.860641 25954 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:35763
I20250905 13:58:53.860782 26104 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:35763 every 8 connection(s)
I20250905 13:58:53.862861 25954 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 13:58:53.860404 25997 raft_consensus.cc:357] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:53.863214 25997 raft_consensus.cc:738] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Initialized, Role: FOLLOWER
I20250905 13:58:53.864009 25997 consensus_queue.cc:260] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:53.871250 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 25954
I20250905 13:58:53.873360 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.194:39317
--local_ip_for_outbound_sockets=127.21.98.194
--tserver_master_addrs=127.21.98.254:41873
--webserver_port=41835
--webserver_interface=127.21.98.194
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:58:53.875679 25997 ts_tablet_manager.cc:1428] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent starting tablet: real 0.034s user 0.026s sys 0.000s
I20250905 13:58:53.877480 25997 tablet_bootstrap.cc:492] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap starting.
I20250905 13:58:53.910226 26105 heartbeater.cc:344] Connected to a master server at 127.21.98.254:41873
I20250905 13:58:53.910668 26105 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:53.911854 26105 heartbeater.cc:507] Master 127.21.98.254:41873 requested a full tablet report, sending...
I20250905 13:58:53.916945 25917 ts_manager.cc:194] Registered new tserver with Master: b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193:35763)
I20250905 13:58:53.921943 25917 catalog_manager.cc:5582] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f reported cstate change: config changed from index -1 to 12, term changed from 0 to 2, VOTER 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) added, VOTER b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) added, VOTER c656231d41e2493f955684515def253f (127.21.98.194) added. New cstate: current_term: 2 committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } } }
I20250905 13:58:54.000056 25917 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:49299
I20250905 13:58:54.004753 26105 heartbeater.cc:499] Master 127.21.98.254:41873 was elected leader, sending a full tablet report...
I20250905 13:58:54.007470 25997 tablet_bootstrap.cc:492] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=250 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:54.009214 25997 tablet_bootstrap.cc:492] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap complete.
I20250905 13:58:54.010914 25997 ts_tablet_manager.cc:1397] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent bootstrapping tablet: real 0.134s user 0.106s sys 0.015s
I20250905 13:58:54.013334 25997 raft_consensus.cc:357] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:54.013870 25997 raft_consensus.cc:738] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Initialized, Role: FOLLOWER
I20250905 13:58:54.014410 25997 consensus_queue.cc:260] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:54.015049 25997 raft_consensus.cc:397] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:58:54.015383 25997 raft_consensus.cc:491] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:58:54.016359 25997 raft_consensus.cc:3058] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Advancing to term 3
I20250905 13:58:54.024804 25997 raft_consensus.cc:513] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:54.025580 25997 leader_election.cc:304] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b41ea60da9974dabb2e4bb44807a7c9f; no voters:
I20250905 13:58:54.026211 25997 leader_election.cc:290] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 election: Requested vote from peers
I20250905 13:58:54.026434 26106 raft_consensus.cc:2802] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 FOLLOWER]: Leader election won for term 3
I20250905 13:58:54.031872 26106 raft_consensus.cc:695] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 LEADER]: Becoming Leader. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Running, Role: LEADER
I20250905 13:58:54.033231 26106 consensus_queue.cc:237] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 7, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } }
I20250905 13:58:54.033679 25997 ts_tablet_manager.cc:1428] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent starting tablet: real 0.022s user 0.015s sys 0.006s
I20250905 13:58:54.034310 25997 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap starting.
I20250905 13:58:54.065959 25916 catalog_manager.cc:5582] T 7c2039b8c6ed49eca474f06b27f83782 P b41ea60da9974dabb2e4bb44807a7c9f reported cstate change: term changed from 2 to 3. New cstate: current_term: 3 leader_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } health_report { overall_health: HEALTHY } } }
I20250905 13:58:54.155642 25997 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:54.156332 25997 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Bootstrap complete.
I20250905 13:58:54.157548 25997 ts_tablet_manager.cc:1397] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent bootstrapping tablet: real 0.123s user 0.086s sys 0.024s
I20250905 13:58:54.159118 25997 raft_consensus.cc:357] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:54.159538 25997 raft_consensus.cc:738] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Initialized, Role: FOLLOWER
I20250905 13:58:54.160045 25997 consensus_queue.cc:260] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:54.161548 25997 ts_tablet_manager.cc:1428] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250905 13:58:54.241649 26110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:54.242086 26110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:54.242506 26110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:54.270390 26110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:54.271166 26110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.194
I20250905 13:58:54.301314 26110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.194:39317
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.98.194
--webserver_port=41835
--tserver_master_addrs=127.21.98.254:41873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.194
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:54.302423 26110 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:54.303881 26110 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:54.315932 26126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:58:55.099735 26132 raft_consensus.cc:491] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:58:55.100389 26132 raft_consensus.cc:513] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:55.120237 26132 leader_election.cc:290] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931), c656231d41e2493f955684515def253f (127.21.98.194:39317)
W20250905 13:58:55.127039 25992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.98.195:43931: connect: Connection refused (error 111)
W20250905 13:58:55.143484 25992 leader_election.cc:336] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): Network error: Client connection negotiation failed: client connection to 127.21.98.195:43931: connect: Connection refused (error 111)
W20250905 13:58:55.144526 25992 leader_election.cc:336] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c656231d41e2493f955684515def253f (127.21.98.194:39317): Network error: Client connection negotiation failed: client connection to 127.21.98.194:39317: connect: Connection refused (error 111)
I20250905 13:58:55.145104 25992 leader_election.cc:304] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b41ea60da9974dabb2e4bb44807a7c9f; no voters: 867e2a35e49a4320b858ce1efad4e408, c656231d41e2493f955684515def253f
I20250905 13:58:55.146080 26132 raft_consensus.cc:2747] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20250905 13:58:54.317299 26127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:55.632272 26128 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1310 milliseconds
W20250905 13:58:55.633488 26110 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.317s user 0.434s sys 0.859s
W20250905 13:58:55.633792 26110 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.317s user 0.434s sys 0.859s
W20250905 13:58:55.633929 26129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:58:55.634047 26110 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:55.635341 26110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:55.637902 26110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:55.639443 26110 hybrid_clock.cc:648] HybridClock initialized: now 1757080735639407 us; error 42 us; skew 500 ppm
I20250905 13:58:55.640317 26110 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:55.648138 26110 webserver.cc:480] Webserver started at http://127.21.98.194:41835/ using document root <none> and password file <none>
I20250905 13:58:55.649248 26110 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:55.649497 26110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:55.657958 26110 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20250905 13:58:55.663511 26140 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:55.664534 26110 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250905 13:58:55.664868 26110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "c656231d41e2493f955684515def253f"
format_stamp: "Formatted at 2025-09-05 13:58:26 on dist-test-slave-g001"
I20250905 13:58:55.666657 26110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:55.744884 26110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:55.746239 26110 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:55.746598 26110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:55.749009 26110 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:55.754133 26147 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250905 13:58:55.764770 26110 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250905 13:58:55.764964 26110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.012s user 0.001s sys 0.001s
I20250905 13:58:55.765197 26110 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250905 13:58:55.770020 26147 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Bootstrap starting.
I20250905 13:58:55.772287 26110 ts_tablet_manager.cc:610] Registered 2 tablets
I20250905 13:58:55.772450 26110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.007s user 0.006s sys 0.000s
I20250905 13:58:55.807402 26162 raft_consensus.cc:491] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:58:55.807989 26162 raft_consensus.cc:513] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:55.810370 26162 leader_election.cc:290] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c656231d41e2493f955684515def253f (127.21.98.194:39317), 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931)
W20250905 13:58:55.825047 25992 leader_election.cc:336] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c656231d41e2493f955684515def253f (127.21.98.194:39317): Network error: Client connection negotiation failed: client connection to 127.21.98.194:39317: connect: Connection refused (error 111)
W20250905 13:58:55.825666 25992 leader_election.cc:336] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): Network error: Client connection negotiation failed: client connection to 127.21.98.195:43931: connect: Connection refused (error 111)
I20250905 13:58:55.826110 25992 leader_election.cc:304] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b41ea60da9974dabb2e4bb44807a7c9f; no voters: 867e2a35e49a4320b858ce1efad4e408, c656231d41e2493f955684515def253f
I20250905 13:58:55.826778 26162 raft_consensus.cc:2747] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 13:58:55.829420 26147 log.cc:826] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:55.927248 26147 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:55.927955 26147 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Bootstrap complete.
I20250905 13:58:55.930115 26147 ts_tablet_manager.cc:1397] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Time spent bootstrapping tablet: real 0.160s user 0.134s sys 0.024s
I20250905 13:58:55.944156 26147 raft_consensus.cc:357] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:55.946624 26147 raft_consensus.cc:738] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c656231d41e2493f955684515def253f, State: Initialized, Role: FOLLOWER
I20250905 13:58:55.947683 26147 consensus_queue.cc:260] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:55.951764 26147 ts_tablet_manager.cc:1428] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f: Time spent starting tablet: real 0.021s user 0.018s sys 0.004s
I20250905 13:58:55.952302 26147 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Bootstrap starting.
I20250905 13:58:55.954126 26110 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.194:39317
I20250905 13:58:55.954560 26259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.194:39317 every 8 connection(s)
I20250905 13:58:55.956620 26110 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 13:58:55.960369 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 26110
I20250905 13:58:55.962050 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.195:43931
--local_ip_for_outbound_sockets=127.21.98.195
--tserver_master_addrs=127.21.98.254:41873
--webserver_port=36707
--webserver_interface=127.21.98.195
--builtin_ntp_servers=127.21.98.212:43167
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20250905 13:58:55.991382 26260 heartbeater.cc:344] Connected to a master server at 127.21.98.254:41873
I20250905 13:58:55.991865 26260 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:55.993078 26260 heartbeater.cc:507] Master 127.21.98.254:41873 requested a full tablet report, sending...
I20250905 13:58:55.997251 25916 ts_manager.cc:194] Registered new tserver with Master: c656231d41e2493f955684515def253f (127.21.98.194:39317)
I20250905 13:58:56.001183 25916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.194:46493
I20250905 13:58:56.065438 26147 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:56.066174 26147 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Bootstrap complete.
I20250905 13:58:56.067368 26147 ts_tablet_manager.cc:1397] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Time spent bootstrapping tablet: real 0.115s user 0.094s sys 0.019s
I20250905 13:58:56.068903 26147 raft_consensus.cc:357] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:56.069335 26147 raft_consensus.cc:738] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c656231d41e2493f955684515def253f, State: Initialized, Role: FOLLOWER
I20250905 13:58:56.069811 26147 consensus_queue.cc:260] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:56.070808 26260 heartbeater.cc:499] Master 127.21.98.254:41873 was elected leader, sending a full tablet report...
I20250905 13:58:56.071349 26147 ts_tablet_manager.cc:1428] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
W20250905 13:58:56.256496 26264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:58:56.257000 26264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:58:56.257452 26264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:58:56.288095 26264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:58:56.288928 26264 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.195
I20250905 13:58:56.321686 26264 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:43167
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.195:43931
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.98.195
--webserver_port=36707
--tserver_master_addrs=127.21.98.254:41873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.195
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:58:56.323005 26264 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:58:56.324532 26264 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:58:56.335880 26271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:58:56.711844 26277 raft_consensus.cc:491] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:58:56.712455 26277 raft_consensus.cc:513] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:56.720048 26277 leader_election.cc:290] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931), c656231d41e2493f955684515def253f (127.21.98.194:39317)
I20250905 13:58:56.735653 26214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8ff2659b4504a0b827dbdd7041eaf50" candidate_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "c656231d41e2493f955684515def253f" is_pre_election: true
I20250905 13:58:56.736518 26214 raft_consensus.cc:2466] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b41ea60da9974dabb2e4bb44807a7c9f in term 2.
W20250905 13:58:56.736698 25992 leader_election.cc:336] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): Network error: Client connection negotiation failed: client connection to 127.21.98.195:43931: connect: Connection refused (error 111)
I20250905 13:58:56.737874 25992 leader_election.cc:304] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b41ea60da9974dabb2e4bb44807a7c9f, c656231d41e2493f955684515def253f; no voters: 867e2a35e49a4320b858ce1efad4e408
I20250905 13:58:56.738483 26277 raft_consensus.cc:2802] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 13:58:56.738782 26277 raft_consensus.cc:491] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 13:58:56.739081 26277 raft_consensus.cc:3058] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Advancing to term 3
I20250905 13:58:56.743031 26277 raft_consensus.cc:513] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:56.744235 26277 leader_election.cc:290] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 election: Requested vote from peers 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931), c656231d41e2493f955684515def253f (127.21.98.194:39317)
I20250905 13:58:56.745904 26214 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b8ff2659b4504a0b827dbdd7041eaf50" candidate_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "c656231d41e2493f955684515def253f"
I20250905 13:58:56.746369 26214 raft_consensus.cc:3058] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Advancing to term 3
W20250905 13:58:56.748157 25992 leader_election.cc:336] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): Network error: Client connection negotiation failed: client connection to 127.21.98.195:43931: connect: Connection refused (error 111)
I20250905 13:58:56.752610 26214 raft_consensus.cc:2466] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b41ea60da9974dabb2e4bb44807a7c9f in term 3.
I20250905 13:58:56.753300 25992 leader_election.cc:304] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b41ea60da9974dabb2e4bb44807a7c9f, c656231d41e2493f955684515def253f; no voters: 867e2a35e49a4320b858ce1efad4e408
I20250905 13:58:56.753995 26277 raft_consensus.cc:2802] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 FOLLOWER]: Leader election won for term 3
I20250905 13:58:56.755887 26277 raft_consensus.cc:695] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 LEADER]: Becoming Leader. State: Replica: b41ea60da9974dabb2e4bb44807a7c9f, State: Running, Role: LEADER
I20250905 13:58:56.756709 26277 consensus_queue.cc:237] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:56.763614 25916 catalog_manager.cc:5582] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f reported cstate change: term changed from 2 to 3, leader changed from 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) to b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193). New cstate: current_term: 3 leader_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 13:58:57.176631 25992 consensus_peers.cc:489] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f -> Peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): Couldn't send request to peer 867e2a35e49a4320b858ce1efad4e408. Status: Network error: Client connection negotiation failed: client connection to 127.21.98.195:43931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 13:58:57.246321 26214 raft_consensus.cc:1273] T b8ff2659b4504a0b827dbdd7041eaf50 P c656231d41e2493f955684515def253f [term 3 FOLLOWER]: Refusing update from remote peer b41ea60da9974dabb2e4bb44807a7c9f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250905 13:58:57.248823 26277 consensus_queue.cc:1035] T b8ff2659b4504a0b827dbdd7041eaf50 P b41ea60da9974dabb2e4bb44807a7c9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.001s
I20250905 13:58:57.262706 26288 raft_consensus.cc:491] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:58:57.263269 26288 raft_consensus.cc:513] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:57.291326 26288 leader_election.cc:290] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193:35763), 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931)
W20250905 13:58:57.308686 26142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.98.195:43931: connect: Connection refused (error 111)
I20250905 13:58:57.316323 26060 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" candidate_uuid: "c656231d41e2493f955684515def253f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" is_pre_election: true
I20250905 13:58:57.317152 26060 raft_consensus.cc:2466] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c656231d41e2493f955684515def253f in term 2.
W20250905 13:58:57.318835 26142 leader_election.cc:336] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): Network error: Client connection negotiation failed: client connection to 127.21.98.195:43931: connect: Connection refused (error 111)
I20250905 13:58:57.320823 26144 leader_election.cc:304] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b41ea60da9974dabb2e4bb44807a7c9f, c656231d41e2493f955684515def253f; no voters: 867e2a35e49a4320b858ce1efad4e408
I20250905 13:58:57.322022 26288 raft_consensus.cc:2802] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 13:58:57.322326 26288 raft_consensus.cc:491] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 13:58:57.322621 26288 raft_consensus.cc:3058] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 2 FOLLOWER]: Advancing to term 3
I20250905 13:58:57.326932 26288 raft_consensus.cc:513] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 3 FOLLOWER]: Starting leader election with config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:57.328274 26288 leader_election.cc:290] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [CANDIDATE]: Term 3 election: Requested vote from peers b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193:35763), 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931)
I20250905 13:58:57.329138 26060 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" candidate_uuid: "c656231d41e2493f955684515def253f" candidate_term: 3 candidate_status { last_received { term: 2 index: 12 } } ignore_live_leader: false dest_uuid: "b41ea60da9974dabb2e4bb44807a7c9f"
I20250905 13:58:57.329702 26060 raft_consensus.cc:3058] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 2 FOLLOWER]: Advancing to term 3
W20250905 13:58:57.332710 26142 leader_election.cc:336] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): Network error: Client connection negotiation failed: client connection to 127.21.98.195:43931: connect: Connection refused (error 111)
I20250905 13:58:57.334817 26060 raft_consensus.cc:2466] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c656231d41e2493f955684515def253f in term 3.
I20250905 13:58:57.335706 26144 leader_election.cc:304] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: b41ea60da9974dabb2e4bb44807a7c9f, c656231d41e2493f955684515def253f; no voters: 867e2a35e49a4320b858ce1efad4e408
I20250905 13:58:57.336269 26288 raft_consensus.cc:2802] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 3 FOLLOWER]: Leader election won for term 3
I20250905 13:58:57.336788 26288 raft_consensus.cc:695] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 3 LEADER]: Becoming Leader. State: Replica: c656231d41e2493f955684515def253f, State: Running, Role: LEADER
I20250905 13:58:57.337646 26288 consensus_queue.cc:237] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:57.345711 25916 catalog_manager.cc:5582] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f reported cstate change: term changed from 2 to 3, leader changed from <none> to c656231d41e2493f955684515def253f (127.21.98.194). New cstate: current_term: 3 leader_uuid: "c656231d41e2493f955684515def253f" committed_config { opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20250905 13:58:56.336381 26272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:56.338817 26274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:58:57.532809 26273 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1194 milliseconds
I20250905 13:58:57.532918 26264 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:58:57.533972 26264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:58:57.536104 26264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:58:57.537484 26264 hybrid_clock.cc:648] HybridClock initialized: now 1757080737537423 us; error 55 us; skew 500 ppm
I20250905 13:58:57.538612 26264 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:58:57.544209 26264 webserver.cc:480] Webserver started at http://127.21.98.195:36707/ using document root <none> and password file <none>
I20250905 13:58:57.545042 26264 fs_manager.cc:362] Metadata directory not provided
I20250905 13:58:57.545219 26264 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:58:57.552102 26264 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20250905 13:58:57.556417 26306 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:58:57.557332 26264 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20250905 13:58:57.557607 26264 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "867e2a35e49a4320b858ce1efad4e408"
format_stamp: "Formatted at 2025-09-05 13:58:28 on dist-test-slave-g001"
I20250905 13:58:57.559420 26264 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:58:57.604632 26264 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:58:57.606233 26264 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:58:57.606676 26264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:58:57.608983 26264 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:58:57.614202 26313 ts_tablet_manager.cc:542] Loading tablet metadata (0/2 complete)
I20250905 13:58:57.625577 26264 ts_tablet_manager.cc:579] Loaded tablet metadata (2 total tablets, 2 live tablets)
I20250905 13:58:57.625775 26264 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.013s user 0.002s sys 0.000s
I20250905 13:58:57.626024 26264 ts_tablet_manager.cc:594] Registering tablets (0/2 complete)
I20250905 13:58:57.630764 26313 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap starting.
I20250905 13:58:57.633481 26264 ts_tablet_manager.cc:610] Registered 2 tablets
I20250905 13:58:57.633648 26264 ts_tablet_manager.cc:589] Time spent register tablets: real 0.008s user 0.005s sys 0.000s
I20250905 13:58:57.686846 26313 log.cc:826] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Log is configured to *not* fsync() on all Append() calls
I20250905 13:58:57.724378 26060 raft_consensus.cc:1273] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 FOLLOWER]: Refusing update from remote peer c656231d41e2493f955684515def253f: Log matching property violated. Preceding OpId in replica: term: 2 index: 12. Preceding OpId from leader: term: 3 index: 13. (index mismatch)
I20250905 13:58:57.726059 26288 consensus_queue.cc:1035] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20250905 13:58:57.760156 26142 consensus_peers.cc:489] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f -> Peer 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): Couldn't send request to peer 867e2a35e49a4320b858ce1efad4e408. Status: Network error: Client connection negotiation failed: client connection to 127.21.98.195:43931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 13:58:57.791996 26214 consensus_queue.cc:237] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 3.13, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:57.797350 26060 raft_consensus.cc:1273] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 FOLLOWER]: Refusing update from remote peer c656231d41e2493f955684515def253f: Log matching property violated. Preceding OpId in replica: term: 3 index: 13. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20250905 13:58:57.798650 26288 consensus_queue.cc:1035] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.001s
I20250905 13:58:57.801769 26313 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:57.802623 26313 tablet_bootstrap.cc:492] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap complete.
I20250905 13:58:57.804467 26313 ts_tablet_manager.cc:1397] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Time spent bootstrapping tablet: real 0.174s user 0.136s sys 0.032s
I20250905 13:58:57.805177 26289 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 3 LEADER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } }
I20250905 13:58:57.810451 26060 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 FOLLOWER]: Committing config change with OpId 3.14: config changed from index 12 to 14, VOTER 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) evicted. New config: { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } }
I20250905 13:58:57.814062 25902 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 2ef4f3179cbd4d71a473c89311a50045 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 13:58:57.818451 25917 catalog_manager.cc:5582] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f reported cstate change: config changed from index 12 to 14, VOTER 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195) evicted. New cstate: current_term: 3 leader_uuid: "c656231d41e2493f955684515def253f" committed_config { opid_index: 14 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20250905 13:58:57.826653 25917 catalog_manager.cc:5774] Failed to send DeleteTablet RPC for tablet 2ef4f3179cbd4d71a473c89311a50045 on TS 867e2a35e49a4320b858ce1efad4e408: Not found: failed to reset TS proxy: Could not find TS for UUID 867e2a35e49a4320b858ce1efad4e408
I20250905 13:58:57.825088 26313 raft_consensus.cc:357] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:57.827994 26313 raft_consensus.cc:738] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867e2a35e49a4320b858ce1efad4e408, State: Initialized, Role: FOLLOWER
I20250905 13:58:57.829074 26313 consensus_queue.cc:260] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } } peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } attrs { promote: false } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } }
I20250905 13:58:57.832028 26214 consensus_queue.cc:237] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 3.14, Last appended by leader: 12, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } }
I20250905 13:58:57.834784 26289 raft_consensus.cc:2953] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f [term 3 LEADER]: Committing config change with OpId 3.15: config changed from index 14 to 15, VOTER b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) evicted. New config: { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } }
I20250905 13:58:57.836730 26313 ts_tablet_manager.cc:1428] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408: Time spent starting tablet: real 0.032s user 0.025s sys 0.004s
I20250905 13:58:57.837634 26313 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap starting.
I20250905 13:58:57.842388 25902 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 2ef4f3179cbd4d71a473c89311a50045 with cas_config_opid_index 14: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 13:58:57.846443 25916 catalog_manager.cc:5582] T 2ef4f3179cbd4d71a473c89311a50045 P c656231d41e2493f955684515def253f reported cstate change: config changed from index 14 to 15, VOTER b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193) evicted. New cstate: current_term: 3 leader_uuid: "c656231d41e2493f955684515def253f" committed_config { opid_index: 15 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } health_report { overall_health: HEALTHY } } }
W20250905 13:58:57.880414 25902 catalog_manager.cc:4726] Async tablet task DeleteTablet RPC for tablet 2ef4f3179cbd4d71a473c89311a50045 on TS 867e2a35e49a4320b858ce1efad4e408 failed: Not found: failed to reset TS proxy: Could not find TS for UUID 867e2a35e49a4320b858ce1efad4e408
I20250905 13:58:57.885008 26040 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ef4f3179cbd4d71a473c89311a50045 with delete_type TABLET_DATA_TOMBSTONED (TS b41ea60da9974dabb2e4bb44807a7c9f not found in new config with opid_index 15) from {username='slave'} at 127.0.0.1:37926
I20250905 13:58:57.888231 26425 tablet_replica.cc:331] stopping tablet replica
I20250905 13:58:57.889559 26425 raft_consensus.cc:2241] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 FOLLOWER]: Raft consensus shutting down.
I20250905 13:58:57.890350 26425 raft_consensus.cc:2270] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f [term 3 FOLLOWER]: Raft consensus is shut down!
I20250905 13:58:57.894623 26425 ts_tablet_manager.cc:1905] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 13:58:57.912304 26264 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.195:43931
I20250905 13:58:57.912457 26428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.195:43931 every 8 connection(s)
I20250905 13:58:57.915169 26425 ts_tablet_manager.cc:1918] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.14
I20250905 13:58:57.915474 26425 log.cc:1199] T 2ef4f3179cbd4d71a473c89311a50045 P b41ea60da9974dabb2e4bb44807a7c9f: Deleting WAL directory at /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/wals/2ef4f3179cbd4d71a473c89311a50045
I20250905 13:58:57.915725 26264 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 13:58:57.917388 25904 catalog_manager.cc:4928] TS b41ea60da9974dabb2e4bb44807a7c9f (127.21.98.193:35763): tablet 2ef4f3179cbd4d71a473c89311a50045 (table TestTable [id=af31f66eacc648f1925e1ef1ec580f02]) successfully deleted
I20250905 13:58:57.925248 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 26264
I20250905 13:58:58.005380 26429 heartbeater.cc:344] Connected to a master server at 127.21.98.254:41873
I20250905 13:58:58.005865 26429 heartbeater.cc:461] Registering TS with master...
I20250905 13:58:58.007038 26429 heartbeater.cc:507] Master 127.21.98.254:41873 requested a full tablet report, sending...
I20250905 13:58:58.011121 25916 ts_manager.cc:194] Registered new tserver with Master: 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931)
I20250905 13:58:58.015116 25916 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.195:42595
I20250905 13:58:58.021184 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:58:58.025657 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:58:58.026861 26313 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=300 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 13:58:58.027500 26313 tablet_bootstrap.cc:492] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Bootstrap complete.
I20250905 13:58:58.028708 26313 ts_tablet_manager.cc:1397] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Time spent bootstrapping tablet: real 0.191s user 0.147s sys 0.027s
W20250905 13:58:58.028942 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
I20250905 13:58:58.030458 26313 raft_consensus.cc:357] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:58.030823 26313 raft_consensus.cc:738] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 867e2a35e49a4320b858ce1efad4e408, State: Initialized, Role: FOLLOWER
I20250905 13:58:58.031301 26313 consensus_queue.cc:260] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 2.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: true peers { permanent_uuid: "c656231d41e2493f955684515def253f" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 39317 } } peers { permanent_uuid: "b41ea60da9974dabb2e4bb44807a7c9f" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 35763 } attrs { promote: false } } peers { permanent_uuid: "867e2a35e49a4320b858ce1efad4e408" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 43931 } attrs { promote: false } }
I20250905 13:58:58.032094 26429 heartbeater.cc:499] Master 127.21.98.254:41873 was elected leader, sending a full tablet report...
I20250905 13:58:58.032490 26313 ts_tablet_manager.cc:1428] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20250905 13:58:58.047451 26356 tablet_service.cc:1515] Processing DeleteTablet for tablet 2ef4f3179cbd4d71a473c89311a50045 with delete_type TABLET_DATA_TOMBSTONED (TS 867e2a35e49a4320b858ce1efad4e408 not found in new config with opid_index 14) from {username='slave'} at 127.0.0.1:38236
I20250905 13:58:58.053251 26436 tablet_replica.cc:331] stopping tablet replica
I20250905 13:58:58.053834 26436 raft_consensus.cc:2241] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 13:58:58.054190 26436 raft_consensus.cc:2270] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 13:58:58.056420 26436 ts_tablet_manager.cc:1905] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 13:58:58.066228 26436 ts_tablet_manager.cc:1918] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.12
I20250905 13:58:58.066476 26436 log.cc:1199] T 2ef4f3179cbd4d71a473c89311a50045 P 867e2a35e49a4320b858ce1efad4e408: Deleting WAL directory at /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/wals/2ef4f3179cbd4d71a473c89311a50045
I20250905 13:58:58.067800 25902 catalog_manager.cc:4928] TS 867e2a35e49a4320b858ce1efad4e408 (127.21.98.195:43931): tablet 2ef4f3179cbd4d71a473c89311a50045 (table TestTable [id=af31f66eacc648f1925e1ef1ec580f02]) successfully deleted
I20250905 13:58:58.260191 26378 raft_consensus.cc:3058] T b8ff2659b4504a0b827dbdd7041eaf50 P 867e2a35e49a4320b858ce1efad4e408 [term 2 FOLLOWER]: Advancing to term 3
W20250905 13:58:59.032655 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:00.036095 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:01.040609 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:01.348256 26101 debug-util.cc:398] Leaking SignalData structure 0x7b08000cb2c0 after lost signal to thread 25975
W20250905 13:59:01.349092 26101 debug-util.cc:398] Leaking SignalData structure 0x7b08000bb940 after lost signal to thread 26104
W20250905 13:59:02.044566 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:03.048195 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:04.051635 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:05.055155 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:06.058367 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:07.061463 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:08.065126 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:09.068339 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:10.071686 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:11.074836 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:12.077903 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:13.081069 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:14.084075 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:15.087460 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:16.090610 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
W20250905 13:59:17.093783 21899 ts_itest-base.cc:209] found only 1 out of 3 replicas of tablet 2ef4f3179cbd4d71a473c89311a50045: tablet_id: "2ef4f3179cbd4d71a473c89311a50045" DEPRECATED_stale: false partition { partition_key_start: "" partition_key_end: "" } interned_replicas { ts_info_idx: 0 role: LEADER }
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/tools/kudu-admin-test.cc:3914: Failure
Failed
Bad status: Not found: not all replicas of tablets comprising table TestTable are registered yet
I20250905 13:59:18.098357 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 25954
I20250905 13:59:18.124393 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 26110
I20250905 13:59:18.149537 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 26264
I20250905 13:59:18.172669 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 25884
2025-09-05T13:59:18Z chronyd exiting
I20250905 13:59:18.217839 21899 test_util.cc:183] -----------------------------------------------
I20250905 13:59:18.218005 21899 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.AdminCliTest.TestRebuildTables.1757080637411871-21899-0
[ FAILED ] AdminCliTest.TestRebuildTables (57468 ms)
[----------] 5 tests from AdminCliTest (120740 ms total)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest
[ RUN ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4
I20250905 13:59:18.221496 21899 test_util.cc:276] Using random seed: 1047493283
I20250905 13:59:18.225383 21899 ts_itest-base.cc:115] Starting cluster with:
I20250905 13:59:18.225520 21899 ts_itest-base.cc:116] --------------
I20250905 13:59:18.225664 21899 ts_itest-base.cc:117] 5 tablet servers
I20250905 13:59:18.225798 21899 ts_itest-base.cc:118] 3 replicas per TS
I20250905 13:59:18.225929 21899 ts_itest-base.cc:119] --------------
2025-09-05T13:59:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T13:59:18Z Disabled control of system clock
I20250905 13:59:18.265280 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:35021
--webserver_interface=127.21.98.254
--webserver_port=0
--builtin_ntp_servers=127.21.98.212:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:35021
--raft_prepare_replacement_before_eviction=true with env {}
W20250905 13:59:18.545645 26461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:18.546094 26461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:18.546448 26461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:18.572999 26461 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 13:59:18.573343 26461 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:59:18.573580 26461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:18.573817 26461 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:59:18.574047 26461 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:59:18.605767 26461 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:35021
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:35021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:18.607041 26461 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:18.608641 26461 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:18.619930 26468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:18.619947 26467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:19.703913 26469 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
W20250905 13:59:19.705417 26470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:59:19.705482 26461 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:19.708279 26461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:19.711301 26461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:19.712728 26461 hybrid_clock.cc:648] HybridClock initialized: now 1757080759712696 us; error 40 us; skew 500 ppm
I20250905 13:59:19.713536 26461 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:19.719650 26461 webserver.cc:480] Webserver started at http://127.21.98.254:38371/ using document root <none> and password file <none>
I20250905 13:59:19.720471 26461 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:19.720672 26461 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:19.721113 26461 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:19.725077 26461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ffa24197764c4bfca2ef8d8e95074b98"
format_stamp: "Formatted at 2025-09-05 13:59:19 on dist-test-slave-g001"
I20250905 13:59:19.726053 26461 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ffa24197764c4bfca2ef8d8e95074b98"
format_stamp: "Formatted at 2025-09-05 13:59:19 on dist-test-slave-g001"
I20250905 13:59:19.732506 26461 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20250905 13:59:19.737787 26477 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:19.738696 26461 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 13:59:19.738978 26461 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ffa24197764c4bfca2ef8d8e95074b98"
format_stamp: "Formatted at 2025-09-05 13:59:19 on dist-test-slave-g001"
I20250905 13:59:19.739231 26461 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:19.792680 26461 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:19.793921 26461 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:19.794301 26461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:19.860733 26461 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:35021
I20250905 13:59:19.860806 26528 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:35021 every 8 connection(s)
I20250905 13:59:19.863242 26461 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 13:59:19.866695 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 26461
I20250905 13:59:19.867210 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 13:59:19.869655 26529 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:59:19.891683 26529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98: Bootstrap starting.
I20250905 13:59:19.896548 26529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98: Neither blocks nor log segments found. Creating new log.
I20250905 13:59:19.897998 26529 log.cc:826] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98: Log is configured to *not* fsync() on all Append() calls
I20250905 13:59:19.901952 26529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98: No bootstrap required, opened a new log
I20250905 13:59:19.917315 26529 raft_consensus.cc:357] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa24197764c4bfca2ef8d8e95074b98" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35021 } }
I20250905 13:59:19.917810 26529 raft_consensus.cc:383] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:59:19.918036 26529 raft_consensus.cc:738] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ffa24197764c4bfca2ef8d8e95074b98, State: Initialized, Role: FOLLOWER
I20250905 13:59:19.918653 26529 consensus_queue.cc:260] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa24197764c4bfca2ef8d8e95074b98" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35021 } }
I20250905 13:59:19.919122 26529 raft_consensus.cc:397] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:59:19.919351 26529 raft_consensus.cc:491] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:59:19.919631 26529 raft_consensus.cc:3058] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:59:19.923434 26529 raft_consensus.cc:513] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa24197764c4bfca2ef8d8e95074b98" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35021 } }
I20250905 13:59:19.923983 26529 leader_election.cc:304] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ffa24197764c4bfca2ef8d8e95074b98; no voters:
I20250905 13:59:19.925386 26529 leader_election.cc:290] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:59:19.925915 26534 raft_consensus.cc:2802] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:59:19.927798 26534 raft_consensus.cc:695] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [term 1 LEADER]: Becoming Leader. State: Replica: ffa24197764c4bfca2ef8d8e95074b98, State: Running, Role: LEADER
I20250905 13:59:19.928550 26534 consensus_queue.cc:237] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa24197764c4bfca2ef8d8e95074b98" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35021 } }
I20250905 13:59:19.928959 26529 sys_catalog.cc:564] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:59:19.940630 26535 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ffa24197764c4bfca2ef8d8e95074b98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa24197764c4bfca2ef8d8e95074b98" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35021 } } }
I20250905 13:59:19.940623 26536 sys_catalog.cc:455] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ffa24197764c4bfca2ef8d8e95074b98. Latest consensus state: current_term: 1 leader_uuid: "ffa24197764c4bfca2ef8d8e95074b98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ffa24197764c4bfca2ef8d8e95074b98" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 35021 } } }
I20250905 13:59:19.941318 26536 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [sys.catalog]: This master's current role is: LEADER
I20250905 13:59:19.941308 26535 sys_catalog.cc:458] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98 [sys.catalog]: This master's current role is: LEADER
I20250905 13:59:19.944635 26542 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:59:19.954422 26542 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:59:19.968925 26542 catalog_manager.cc:1349] Generated new cluster ID: 90e63f2743c1483e8e9b68ac7eb34496
I20250905 13:59:19.969192 26542 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:59:19.989771 26542 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 13:59:19.991061 26542 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:59:20.004271 26542 catalog_manager.cc:5955] T 00000000000000000000000000000000 P ffa24197764c4bfca2ef8d8e95074b98: Generated new TSK 0
I20250905 13:59:20.005012 26542 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 13:59:20.016922 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:0
--local_ip_for_outbound_sockets=127.21.98.193
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35021
--builtin_ntp_servers=127.21.98.212:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20250905 13:59:20.303208 26553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:20.303659 26553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:20.304108 26553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:20.331912 26553 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 13:59:20.332415 26553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:20.333142 26553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 13:59:20.364836 26553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:20.366133 26553 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:20.367707 26553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:20.379570 26559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:20.380861 26560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:21.752662 26562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:21.756264 26561 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1373 milliseconds
W20250905 13:59:21.756883 26553 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.376s user 0.474s sys 0.899s
W20250905 13:59:21.757181 26553 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.376s user 0.474s sys 0.899s
I20250905 13:59:21.757448 26553 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:21.758476 26553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:21.761163 26553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:21.762631 26553 hybrid_clock.cc:648] HybridClock initialized: now 1757080761762561 us; error 77 us; skew 500 ppm
I20250905 13:59:21.763401 26553 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:21.770727 26553 webserver.cc:480] Webserver started at http://127.21.98.193:36839/ using document root <none> and password file <none>
I20250905 13:59:21.771718 26553 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:21.771975 26553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:21.772432 26553 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:21.776679 26553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "fad603213ca24885945180c1f6284eb9"
format_stamp: "Formatted at 2025-09-05 13:59:21 on dist-test-slave-g001"
I20250905 13:59:21.777813 26553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "fad603213ca24885945180c1f6284eb9"
format_stamp: "Formatted at 2025-09-05 13:59:21 on dist-test-slave-g001"
I20250905 13:59:21.785163 26553 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.005s
I20250905 13:59:21.791194 26569 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:21.792294 26553 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250905 13:59:21.792572 26553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "fad603213ca24885945180c1f6284eb9"
format_stamp: "Formatted at 2025-09-05 13:59:21 on dist-test-slave-g001"
I20250905 13:59:21.792928 26553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:21.860545 26553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:21.862334 26553 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:21.862799 26553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:21.875994 26553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:59:21.880290 26553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:59:21.880465 26553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:21.880717 26553 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:59:21.880875 26553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:22.034876 26553 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:37881
I20250905 13:59:22.035072 26681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:37881 every 8 connection(s)
I20250905 13:59:22.037330 26553 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 13:59:22.040330 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 26553
I20250905 13:59:22.040870 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 13:59:22.049047 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.194:0
--local_ip_for_outbound_sockets=127.21.98.194
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35021
--builtin_ntp_servers=127.21.98.212:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250905 13:59:22.085763 26682 heartbeater.cc:344] Connected to a master server at 127.21.98.254:35021
I20250905 13:59:22.086156 26682 heartbeater.cc:461] Registering TS with master...
I20250905 13:59:22.087159 26682 heartbeater.cc:507] Master 127.21.98.254:35021 requested a full tablet report, sending...
I20250905 13:59:22.089355 26494 ts_manager.cc:194] Registered new tserver with Master: fad603213ca24885945180c1f6284eb9 (127.21.98.193:37881)
I20250905 13:59:22.091176 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:39987
W20250905 13:59:22.359326 26686 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:22.359839 26686 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:22.360332 26686 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:22.390329 26686 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 13:59:22.390736 26686 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:22.391614 26686 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.194
I20250905 13:59:22.425321 26686 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.194
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:22.426561 26686 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:22.428010 26686 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:22.440150 26692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:59:23.094132 26682 heartbeater.cc:499] Master 127.21.98.254:35021 was elected leader, sending a full tablet report...
W20250905 13:59:22.440574 26693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:23.708909 26695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:23.711591 26694 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1266 milliseconds
W20250905 13:59:23.712316 26686 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.272s user 0.387s sys 0.871s
W20250905 13:59:23.712659 26686 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.273s user 0.387s sys 0.871s
I20250905 13:59:23.712947 26686 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:23.714371 26686 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:23.717007 26686 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:23.718545 26686 hybrid_clock.cc:648] HybridClock initialized: now 1757080763718503 us; error 36 us; skew 500 ppm
I20250905 13:59:23.719615 26686 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:23.727840 26686 webserver.cc:480] Webserver started at http://127.21.98.194:45519/ using document root <none> and password file <none>
I20250905 13:59:23.729195 26686 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:23.729475 26686 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:23.730023 26686 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:23.736918 26686 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "b2f02f1dfb704d46907d4dd9ec968ddf"
format_stamp: "Formatted at 2025-09-05 13:59:23 on dist-test-slave-g001"
I20250905 13:59:23.738495 26686 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "b2f02f1dfb704d46907d4dd9ec968ddf"
format_stamp: "Formatted at 2025-09-05 13:59:23 on dist-test-slave-g001"
I20250905 13:59:23.747920 26686 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.005s sys 0.002s
I20250905 13:59:23.755142 26702 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:23.756243 26686 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20250905 13:59:23.756634 26686 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "b2f02f1dfb704d46907d4dd9ec968ddf"
format_stamp: "Formatted at 2025-09-05 13:59:23 on dist-test-slave-g001"
I20250905 13:59:23.757068 26686 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:23.835178 26686 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:23.836503 26686 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:23.836876 26686 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:23.839104 26686 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:59:23.842806 26686 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:59:23.843022 26686 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20250905 13:59:23.843197 26686 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:59:23.843364 26686 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:23.967933 26686 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.194:38163
I20250905 13:59:23.968029 26814 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.194:38163 every 8 connection(s)
I20250905 13:59:23.970356 26686 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20250905 13:59:23.977365 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 26686
I20250905 13:59:23.977737 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20250905 13:59:23.983167 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.195:0
--local_ip_for_outbound_sockets=127.21.98.195
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35021
--builtin_ntp_servers=127.21.98.212:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250905 13:59:23.989863 26815 heartbeater.cc:344] Connected to a master server at 127.21.98.254:35021
I20250905 13:59:23.990252 26815 heartbeater.cc:461] Registering TS with master...
I20250905 13:59:23.991204 26815 heartbeater.cc:507] Master 127.21.98.254:35021 requested a full tablet report, sending...
I20250905 13:59:23.993250 26494 ts_manager.cc:194] Registered new tserver with Master: b2f02f1dfb704d46907d4dd9ec968ddf (127.21.98.194:38163)
I20250905 13:59:23.994349 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.194:59785
W20250905 13:59:24.264027 26819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:24.264465 26819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:24.264894 26819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:24.291497 26819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 13:59:24.291859 26819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:24.292630 26819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.195
I20250905 13:59:24.324167 26819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.195
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:24.325408 26819 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:24.327047 26819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:24.337576 26825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:59:24.997691 26815 heartbeater.cc:499] Master 127.21.98.254:35021 was elected leader, sending a full tablet report...
W20250905 13:59:24.339130 26826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:25.742244 26824 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 26819
W20250905 13:59:25.969918 26819 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.631s user 0.608s sys 1.022s
W20250905 13:59:25.971261 26819 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.633s user 0.608s sys 1.023s
W20250905 13:59:25.972914 26828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:25.975283 26827 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1632 milliseconds
I20250905 13:59:25.975306 26819 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:25.976758 26819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:25.979216 26819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:25.980655 26819 hybrid_clock.cc:648] HybridClock initialized: now 1757080765980620 us; error 27 us; skew 500 ppm
I20250905 13:59:25.981685 26819 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:25.987612 26819 webserver.cc:480] Webserver started at http://127.21.98.195:46623/ using document root <none> and password file <none>
I20250905 13:59:25.988432 26819 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:25.988590 26819 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:25.988916 26819 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:25.992996 26819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "511afeff9d6b4c89bfe7ebe07d3b72c4"
format_stamp: "Formatted at 2025-09-05 13:59:25 on dist-test-slave-g001"
I20250905 13:59:25.993902 26819 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "511afeff9d6b4c89bfe7ebe07d3b72c4"
format_stamp: "Formatted at 2025-09-05 13:59:25 on dist-test-slave-g001"
I20250905 13:59:26.000355 26819 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20250905 13:59:26.005928 26835 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:26.006803 26819 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 13:59:26.007117 26819 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "511afeff9d6b4c89bfe7ebe07d3b72c4"
format_stamp: "Formatted at 2025-09-05 13:59:25 on dist-test-slave-g001"
I20250905 13:59:26.007392 26819 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:26.068804 26819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:26.070408 26819 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:26.070807 26819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:26.073619 26819 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:59:26.077494 26819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:59:26.077663 26819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:26.077826 26819 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:59:26.077939 26819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:26.205152 26819 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.195:36681
I20250905 13:59:26.205252 26947 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.195:36681 every 8 connection(s)
I20250905 13:59:26.207536 26819 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20250905 13:59:26.213845 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 26819
I20250905 13:59:26.214331 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20250905 13:59:26.219967 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.196:0
--local_ip_for_outbound_sockets=127.21.98.196
--webserver_interface=127.21.98.196
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35021
--builtin_ntp_servers=127.21.98.212:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250905 13:59:26.227092 26948 heartbeater.cc:344] Connected to a master server at 127.21.98.254:35021
I20250905 13:59:26.227488 26948 heartbeater.cc:461] Registering TS with master...
I20250905 13:59:26.228447 26948 heartbeater.cc:507] Master 127.21.98.254:35021 requested a full tablet report, sending...
I20250905 13:59:26.230583 26494 ts_manager.cc:194] Registered new tserver with Master: 511afeff9d6b4c89bfe7ebe07d3b72c4 (127.21.98.195:36681)
I20250905 13:59:26.231918 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.195:37597
W20250905 13:59:26.505645 26952 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:26.506168 26952 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:26.506570 26952 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:26.535400 26952 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 13:59:26.535748 26952 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:26.536541 26952 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.196
I20250905 13:59:26.566309 26952 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.196:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.21.98.196
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.196
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:26.567617 26952 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:26.568997 26952 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:26.580780 26958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:59:27.234793 26948 heartbeater.cc:499] Master 127.21.98.254:35021 was elected leader, sending a full tablet report...
W20250905 13:59:26.581388 26959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:27.710506 26961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:27.713991 26960 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1126 milliseconds
W20250905 13:59:27.714646 26952 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.133s user 0.389s sys 0.738s
W20250905 13:59:27.714902 26952 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.134s user 0.389s sys 0.738s
I20250905 13:59:27.715094 26952 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:27.716106 26952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:27.718250 26952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:27.719570 26952 hybrid_clock.cc:648] HybridClock initialized: now 1757080767719536 us; error 33 us; skew 500 ppm
I20250905 13:59:27.720345 26952 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:27.727169 26952 webserver.cc:480] Webserver started at http://127.21.98.196:34885/ using document root <none> and password file <none>
I20250905 13:59:27.728127 26952 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:27.728332 26952 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:27.728730 26952 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:27.732909 26952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "88c1f6aaa5f5431b83f7267010c6c433"
format_stamp: "Formatted at 2025-09-05 13:59:27 on dist-test-slave-g001"
I20250905 13:59:27.733894 26952 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "88c1f6aaa5f5431b83f7267010c6c433"
format_stamp: "Formatted at 2025-09-05 13:59:27 on dist-test-slave-g001"
I20250905 13:59:27.741393 26952 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.005s
I20250905 13:59:27.746978 26968 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:27.748131 26952 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250905 13:59:27.748523 26952 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "88c1f6aaa5f5431b83f7267010c6c433"
format_stamp: "Formatted at 2025-09-05 13:59:27 on dist-test-slave-g001"
I20250905 13:59:27.748822 26952 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:27.828989 26952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:27.830346 26952 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:27.830767 26952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:27.833189 26952 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:59:27.836849 26952 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:59:27.837057 26952 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:27.837307 26952 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:59:27.837463 26952 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:27.966249 26952 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.196:35831
I20250905 13:59:27.966387 27081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.196:35831 every 8 connection(s)
I20250905 13:59:27.969636 26952 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20250905 13:59:27.978776 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 26952
I20250905 13:59:27.979972 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20250905 13:59:27.995718 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.197:0
--local_ip_for_outbound_sockets=127.21.98.197
--webserver_interface=127.21.98.197
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35021
--builtin_ntp_servers=127.21.98.212:37595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20250905 13:59:28.009891 27082 heartbeater.cc:344] Connected to a master server at 127.21.98.254:35021
I20250905 13:59:28.010466 27082 heartbeater.cc:461] Registering TS with master...
I20250905 13:59:28.011897 27082 heartbeater.cc:507] Master 127.21.98.254:35021 requested a full tablet report, sending...
I20250905 13:59:28.014254 26494 ts_manager.cc:194] Registered new tserver with Master: 88c1f6aaa5f5431b83f7267010c6c433 (127.21.98.196:35831)
I20250905 13:59:28.015372 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.196:50303
W20250905 13:59:28.288542 27086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:28.288964 27086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:28.289487 27086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:28.319649 27086 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20250905 13:59:28.320025 27086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:28.320721 27086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.197
I20250905 13:59:28.351431 27086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:37595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.197:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.21.98.197
--webserver_port=0
--tserver_master_addrs=127.21.98.254:35021
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.197
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:28.352736 27086 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:28.354220 27086 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:28.364986 27092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:59:29.018869 27082 heartbeater.cc:499] Master 127.21.98.254:35021 was elected leader, sending a full tablet report...
W20250905 13:59:28.365383 27093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:29.512127 27095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:29.514669 27094 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1144 milliseconds
W20250905 13:59:29.514803 27086 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.149s user 0.437s sys 0.709s
W20250905 13:59:29.515138 27086 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.150s user 0.438s sys 0.709s
I20250905 13:59:29.515353 27086 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:29.516309 27086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:29.518280 27086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:29.519584 27086 hybrid_clock.cc:648] HybridClock initialized: now 1757080769519556 us; error 29 us; skew 500 ppm
I20250905 13:59:29.520332 27086 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:29.527352 27086 webserver.cc:480] Webserver started at http://127.21.98.197:38683/ using document root <none> and password file <none>
I20250905 13:59:29.528299 27086 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:29.528503 27086 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:29.528895 27086 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:29.533164 27086 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "9d33399a7a0b4d819a06a95d2da7ab4d"
format_stamp: "Formatted at 2025-09-05 13:59:29 on dist-test-slave-g001"
I20250905 13:59:29.534097 27086 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "9d33399a7a0b4d819a06a95d2da7ab4d"
format_stamp: "Formatted at 2025-09-05 13:59:29 on dist-test-slave-g001"
I20250905 13:59:29.541328 27086 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.005s
I20250905 13:59:29.546801 27103 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:29.547853 27086 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20250905 13:59:29.548125 27086 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "9d33399a7a0b4d819a06a95d2da7ab4d"
format_stamp: "Formatted at 2025-09-05 13:59:29 on dist-test-slave-g001"
I20250905 13:59:29.548422 27086 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:29.606132 27086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:29.607414 27086 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:29.607770 27086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:29.609880 27086 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:59:29.613688 27086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:59:29.613863 27086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:29.614091 27086 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:59:29.614264 27086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:29.740269 27086 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.197:44727
I20250905 13:59:29.740360 27215 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.197:44727 every 8 connection(s)
I20250905 13:59:29.742673 27086 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20250905 13:59:29.745093 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 27086
I20250905 13:59:29.745530 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.EnableKudu1097AndDownTS_MoveTabletParamTest.Test_4.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20250905 13:59:29.761876 27216 heartbeater.cc:344] Connected to a master server at 127.21.98.254:35021
I20250905 13:59:29.762245 27216 heartbeater.cc:461] Registering TS with master...
I20250905 13:59:29.763186 27216 heartbeater.cc:507] Master 127.21.98.254:35021 requested a full tablet report, sending...
I20250905 13:59:29.765285 26494 ts_manager.cc:194] Registered new tserver with Master: 9d33399a7a0b4d819a06a95d2da7ab4d (127.21.98.197:44727)
I20250905 13:59:29.766530 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.197:33353
I20250905 13:59:29.779850 21899 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250905 13:59:29.813406 26494 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58158:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250905 13:59:29.902150 27017 tablet_service.cc:1468] Processing CreateTablet for tablet 577160f9453c41c0b2b2f43c52edf378 (DEFAULT_TABLE table=TestTable [id=6bc73d3c529d44788c967a377fa1089f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:59:29.902890 27151 tablet_service.cc:1468] Processing CreateTablet for tablet 577160f9453c41c0b2b2f43c52edf378 (DEFAULT_TABLE table=TestTable [id=6bc73d3c529d44788c967a377fa1089f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:59:29.904198 27017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 577160f9453c41c0b2b2f43c52edf378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:59:29.903918 26750 tablet_service.cc:1468] Processing CreateTablet for tablet 577160f9453c41c0b2b2f43c52edf378 (DEFAULT_TABLE table=TestTable [id=6bc73d3c529d44788c967a377fa1089f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:59:29.904656 27151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 577160f9453c41c0b2b2f43c52edf378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:59:29.905512 26750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 577160f9453c41c0b2b2f43c52edf378. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:59:29.936601 27235 tablet_bootstrap.cc:492] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433: Bootstrap starting.
I20250905 13:59:29.936983 27236 tablet_bootstrap.cc:492] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf: Bootstrap starting.
I20250905 13:59:29.940379 27237 tablet_bootstrap.cc:492] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d: Bootstrap starting.
I20250905 13:59:29.946166 27235 tablet_bootstrap.cc:654] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433: Neither blocks nor log segments found. Creating new log.
I20250905 13:59:29.947239 27236 tablet_bootstrap.cc:654] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf: Neither blocks nor log segments found. Creating new log.
I20250905 13:59:29.947751 27237 tablet_bootstrap.cc:654] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d: Neither blocks nor log segments found. Creating new log.
I20250905 13:59:29.948691 27235 log.cc:826] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433: Log is configured to *not* fsync() on all Append() calls
I20250905 13:59:29.949517 27236 log.cc:826] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf: Log is configured to *not* fsync() on all Append() calls
I20250905 13:59:29.949936 27237 log.cc:826] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d: Log is configured to *not* fsync() on all Append() calls
I20250905 13:59:29.958689 27237 tablet_bootstrap.cc:492] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d: No bootstrap required, opened a new log
I20250905 13:59:29.959349 27237 ts_tablet_manager.cc:1397] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d: Time spent bootstrapping tablet: real 0.020s user 0.013s sys 0.005s
I20250905 13:59:29.959491 27235 tablet_bootstrap.cc:492] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433: No bootstrap required, opened a new log
I20250905 13:59:29.960027 27235 ts_tablet_manager.cc:1397] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433: Time spent bootstrapping tablet: real 0.024s user 0.016s sys 0.005s
I20250905 13:59:29.963521 27236 tablet_bootstrap.cc:492] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf: No bootstrap required, opened a new log
I20250905 13:59:29.964015 27236 ts_tablet_manager.cc:1397] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf: Time spent bootstrapping tablet: real 0.028s user 0.015s sys 0.009s
I20250905 13:59:29.983698 27237 raft_consensus.cc:357] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 38163 } } peers { permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 } } peers { permanent_uuid: "88c1f6aaa5f5431b83f7267010c6c433" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 35831 } }
I20250905 13:59:29.984622 27237 raft_consensus.cc:383] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:59:29.984939 27237 raft_consensus.cc:738] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d33399a7a0b4d819a06a95d2da7ab4d, State: Initialized, Role: FOLLOWER
I20250905 13:59:29.985903 27237 consensus_queue.cc:260] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 38163 } } peers { permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 } } peers { permanent_uuid: "88c1f6aaa5f5431b83f7267010c6c433" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 35831 } }
I20250905 13:59:29.990108 27236 raft_consensus.cc:357] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 38163 } } peers { permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 } } peers { permanent_uuid: "88c1f6aaa5f5431b83f7267010c6c433" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 35831 } }
I20250905 13:59:29.991163 27236 raft_consensus.cc:383] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:59:29.991582 27236 raft_consensus.cc:738] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2f02f1dfb704d46907d4dd9ec968ddf, State: Initialized, Role: FOLLOWER
I20250905 13:59:29.992690 27236 consensus_queue.cc:260] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 38163 } } peers { permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 } } peers { permanent_uuid: "88c1f6aaa5f5431b83f7267010c6c433" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 35831 } }
I20250905 13:59:29.996654 27237 ts_tablet_manager.cc:1428] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d: Time spent starting tablet: real 0.037s user 0.029s sys 0.005s
W20250905 13:59:29.997494 27217 tablet.cc:2378] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:59:29.997841 27216 heartbeater.cc:499] Master 127.21.98.254:35021 was elected leader, sending a full tablet report...
I20250905 13:59:30.002401 27236 ts_tablet_manager.cc:1428] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf: Time spent starting tablet: real 0.038s user 0.030s sys 0.004s
I20250905 13:59:30.011605 27235 raft_consensus.cc:357] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 38163 } } peers { permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 } } peers { permanent_uuid: "88c1f6aaa5f5431b83f7267010c6c433" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 35831 } }
I20250905 13:59:30.012631 27235 raft_consensus.cc:383] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:59:30.012950 27235 raft_consensus.cc:738] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88c1f6aaa5f5431b83f7267010c6c433, State: Initialized, Role: FOLLOWER
I20250905 13:59:30.013895 27235 consensus_queue.cc:260] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 38163 } } peers { permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 } } peers { permanent_uuid: "88c1f6aaa5f5431b83f7267010c6c433" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 35831 } }
I20250905 13:59:30.022656 27235 ts_tablet_manager.cc:1428] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433: Time spent starting tablet: real 0.062s user 0.032s sys 0.003s
I20250905 13:59:30.083436 27243 raft_consensus.cc:491] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:59:30.083807 27243 raft_consensus.cc:513] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 38163 } } peers { permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 } } peers { permanent_uuid: "88c1f6aaa5f5431b83f7267010c6c433" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 35831 } }
I20250905 13:59:30.086014 27243 leader_election.cc:290] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2f02f1dfb704d46907d4dd9ec968ddf (127.21.98.194:38163), 9d33399a7a0b4d819a06a95d2da7ab4d (127.21.98.197:44727)
I20250905 13:59:30.095861 27171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "577160f9453c41c0b2b2f43c52edf378" candidate_uuid: "88c1f6aaa5f5431b83f7267010c6c433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" is_pre_election: true
I20250905 13:59:30.095862 26770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "577160f9453c41c0b2b2f43c52edf378" candidate_uuid: "88c1f6aaa5f5431b83f7267010c6c433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" is_pre_election: true
I20250905 13:59:30.096525 27171 raft_consensus.cc:2466] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88c1f6aaa5f5431b83f7267010c6c433 in term 0.
I20250905 13:59:30.096520 26770 raft_consensus.cc:2466] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 88c1f6aaa5f5431b83f7267010c6c433 in term 0.
I20250905 13:59:30.097563 26972 leader_election.cc:304] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88c1f6aaa5f5431b83f7267010c6c433, b2f02f1dfb704d46907d4dd9ec968ddf; no voters:
I20250905 13:59:30.098124 27243 raft_consensus.cc:2802] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 13:59:30.098450 27243 raft_consensus.cc:491] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 13:59:30.098649 27243 raft_consensus.cc:3058] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:59:30.102617 27243 raft_consensus.cc:513] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 38163 } } peers { permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 } } peers { permanent_uuid: "88c1f6aaa5f5431b83f7267010c6c433" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 35831 } }
I20250905 13:59:30.103956 27243 leader_election.cc:290] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [CANDIDATE]: Term 1 election: Requested vote from peers b2f02f1dfb704d46907d4dd9ec968ddf (127.21.98.194:38163), 9d33399a7a0b4d819a06a95d2da7ab4d (127.21.98.197:44727)
I20250905 13:59:30.104521 26770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "577160f9453c41c0b2b2f43c52edf378" candidate_uuid: "88c1f6aaa5f5431b83f7267010c6c433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf"
I20250905 13:59:30.104787 27171 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "577160f9453c41c0b2b2f43c52edf378" candidate_uuid: "88c1f6aaa5f5431b83f7267010c6c433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d"
I20250905 13:59:30.104907 26770 raft_consensus.cc:3058] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:59:30.105183 27171 raft_consensus.cc:3058] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:59:30.109107 26770 raft_consensus.cc:2466] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88c1f6aaa5f5431b83f7267010c6c433 in term 1.
I20250905 13:59:30.109473 27171 raft_consensus.cc:2466] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 88c1f6aaa5f5431b83f7267010c6c433 in term 1.
I20250905 13:59:30.109905 26972 leader_election.cc:304] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88c1f6aaa5f5431b83f7267010c6c433, b2f02f1dfb704d46907d4dd9ec968ddf; no voters:
I20250905 13:59:30.110481 27243 raft_consensus.cc:2802] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:59:30.112185 27243 raft_consensus.cc:695] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [term 1 LEADER]: Becoming Leader. State: Replica: 88c1f6aaa5f5431b83f7267010c6c433, State: Running, Role: LEADER
I20250905 13:59:30.113049 27243 consensus_queue.cc:237] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 38163 } } peers { permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 } } peers { permanent_uuid: "88c1f6aaa5f5431b83f7267010c6c433" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 35831 } }
I20250905 13:59:30.121428 26492 catalog_manager.cc:5582] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 reported cstate change: term changed from 0 to 1, leader changed from <none> to 88c1f6aaa5f5431b83f7267010c6c433 (127.21.98.196). New cstate: current_term: 1 leader_uuid: "88c1f6aaa5f5431b83f7267010c6c433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2f02f1dfb704d46907d4dd9ec968ddf" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 38163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "88c1f6aaa5f5431b83f7267010c6c433" member_type: VOTER last_known_addr { host: "127.21.98.196" port: 35831 } health_report { overall_health: HEALTHY } } }
I20250905 13:59:30.178969 21899 external_mini_cluster.cc:949] 5 TS(s) registered with all masters
I20250905 13:59:30.181856 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver b2f02f1dfb704d46907d4dd9ec968ddf to finish bootstrapping
I20250905 13:59:30.193611 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 88c1f6aaa5f5431b83f7267010c6c433 to finish bootstrapping
I20250905 13:59:30.203893 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 9d33399a7a0b4d819a06a95d2da7ab4d to finish bootstrapping
I20250905 13:59:30.214820 21899 test_util.cc:276] Using random seed: 1059486612
W20250905 13:59:30.231622 26816 tablet.cc:2378] T 577160f9453c41c0b2b2f43c52edf378 P b2f02f1dfb704d46907d4dd9ec968ddf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:59:30.237738 21899 test_workload.cc:405] TestWorkload: Skipping table creation because table TestTable already exists
I20250905 13:59:30.238468 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 26686
W20250905 13:59:30.242511 27083 tablet.cc:2378] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:59:30.267951 27171 raft_consensus.cc:1273] T 577160f9453c41c0b2b2f43c52edf378 P 9d33399a7a0b4d819a06a95d2da7ab4d [term 1 FOLLOWER]: Refusing update from remote peer 88c1f6aaa5f5431b83f7267010c6c433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250905 13:59:30.268102 26972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.98.194:38163: connect: Connection refused (error 111)
I20250905 13:59:30.269861 27243 consensus_queue.cc:1035] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d33399a7a0b4d819a06a95d2da7ab4d" member_type: VOTER last_known_addr { host: "127.21.98.197" port: 44727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 13:59:30.277933 26972 consensus_peers.cc:489] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 -> Peer b2f02f1dfb704d46907d4dd9ec968ddf (127.21.98.194:38163): Couldn't send request to peer b2f02f1dfb704d46907d4dd9ec968ddf. Status: Network error: Client connection negotiation failed: client connection to 127.21.98.194:38163: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 13:59:30.287273 27256 mvcc.cc:204] Tried to move back new op lower bound from 7197002835002253312 to 7197002834389929984. Current Snapshot: MvccSnapshot[applied={T|T < 7197002835002253312}]
I20250905 13:59:30.295676 27258 mvcc.cc:204] Tried to move back new op lower bound from 7197002835002253312 to 7197002834389929984. Current Snapshot: MvccSnapshot[applied={T|T < 7197002835002253312}]
W20250905 13:59:32.722309 26972 consensus_peers.cc:489] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 -> Peer b2f02f1dfb704d46907d4dd9ec968ddf (127.21.98.194:38163): Couldn't send request to peer b2f02f1dfb704d46907d4dd9ec968ddf. Status: Network error: Client connection negotiation failed: client connection to 127.21.98.194:38163: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250905 13:59:32.806627 26883 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 13:59:32.845988 27151 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 13:59:32.855682 27017 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 13:59:32.897251 26617 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 13:59:34.569205 26883 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 13:59:34.580377 27017 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 13:59:34.608259 27151 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 13:59:34.621694 26617 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
W20250905 13:59:35.531594 26972 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.21.98.194:38163: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20250905 13:59:35.534417 26972 consensus_peers.cc:489] T 577160f9453c41c0b2b2f43c52edf378 P 88c1f6aaa5f5431b83f7267010c6c433 -> Peer b2f02f1dfb704d46907d4dd9ec968ddf (127.21.98.194:38163): Couldn't send request to peer b2f02f1dfb704d46907d4dd9ec968ddf. Status: Network error: Client connection negotiation failed: client connection to 127.21.98.194:38163: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250905 13:59:36.987783 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 26553
I20250905 13:59:37.013154 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 26819
I20250905 13:59:37.036667 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 26952
I20250905 13:59:37.072989 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 27086
I20250905 13:59:37.105566 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 26461
2025-09-05T13:59:37Z chronyd exiting
[ OK ] EnableKudu1097AndDownTS/MoveTabletParamTest.Test/4 (18937 ms)
[----------] 1 test from EnableKudu1097AndDownTS/MoveTabletParamTest (18937 ms total)
[----------] 1 test from ListTableCliSimpleParamTest
[ RUN ] ListTableCliSimpleParamTest.TestListTables/2
I20250905 13:59:37.159015 21899 test_util.cc:276] Using random seed: 1066430800
I20250905 13:59:37.162515 21899 ts_itest-base.cc:115] Starting cluster with:
I20250905 13:59:37.162659 21899 ts_itest-base.cc:116] --------------
I20250905 13:59:37.162825 21899 ts_itest-base.cc:117] 1 tablet servers
I20250905 13:59:37.163029 21899 ts_itest-base.cc:118] 1 replicas per TS
I20250905 13:59:37.163187 21899 ts_itest-base.cc:119] --------------
2025-09-05T13:59:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T13:59:37Z Disabled control of system clock
I20250905 13:59:37.203387 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:44659
--webserver_interface=127.21.98.254
--webserver_port=0
--builtin_ntp_servers=127.21.98.212:37663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:44659 with env {}
W20250905 13:59:37.483685 27360 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:37.484143 27360 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:37.484495 27360 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:37.513110 27360 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:59:37.513378 27360 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:37.513553 27360 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:59:37.513718 27360 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:59:37.544361 27360 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:37663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:44659
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:44659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:37.545470 27360 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:37.546854 27360 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:37.556780 27366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:37.557071 27367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:38.658990 27369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:38.661936 27368 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1100 milliseconds
W20250905 13:59:38.662524 27360 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.106s user 0.371s sys 0.730s
W20250905 13:59:38.662804 27360 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.106s user 0.371s sys 0.730s
I20250905 13:59:38.663060 27360 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:38.664103 27360 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:38.666496 27360 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:38.667860 27360 hybrid_clock.cc:648] HybridClock initialized: now 1757080778667820 us; error 56 us; skew 500 ppm
I20250905 13:59:38.668628 27360 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:38.675633 27360 webserver.cc:480] Webserver started at http://127.21.98.254:45281/ using document root <none> and password file <none>
I20250905 13:59:38.676708 27360 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:38.676944 27360 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:38.677376 27360 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:38.682803 27360 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "de61f9efed4348729d76fc670fded41d"
format_stamp: "Formatted at 2025-09-05 13:59:38 on dist-test-slave-g001"
I20250905 13:59:38.683903 27360 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "de61f9efed4348729d76fc670fded41d"
format_stamp: "Formatted at 2025-09-05 13:59:38 on dist-test-slave-g001"
I20250905 13:59:38.691242 27360 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20250905 13:59:38.697116 27377 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:38.698232 27360 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20250905 13:59:38.698539 27360 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
uuid: "de61f9efed4348729d76fc670fded41d"
format_stamp: "Formatted at 2025-09-05 13:59:38 on dist-test-slave-g001"
I20250905 13:59:38.698918 27360 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:38.772033 27360 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:38.773407 27360 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:38.773821 27360 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:38.836306 27360 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:44659
I20250905 13:59:38.836385 27428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:44659 every 8 connection(s)
I20250905 13:59:38.838953 27360 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20250905 13:59:38.844341 27429 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:59:38.849086 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 27360
I20250905 13:59:38.849576 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/master-0/wal/instance
I20250905 13:59:38.863319 27429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d: Bootstrap starting.
I20250905 13:59:38.869203 27429 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d: Neither blocks nor log segments found. Creating new log.
I20250905 13:59:38.871366 27429 log.cc:826] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d: Log is configured to *not* fsync() on all Append() calls
I20250905 13:59:38.875749 27429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d: No bootstrap required, opened a new log
I20250905 13:59:38.890739 27429 raft_consensus.cc:357] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de61f9efed4348729d76fc670fded41d" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 44659 } }
I20250905 13:59:38.891292 27429 raft_consensus.cc:383] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:59:38.891486 27429 raft_consensus.cc:738] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de61f9efed4348729d76fc670fded41d, State: Initialized, Role: FOLLOWER
I20250905 13:59:38.892035 27429 consensus_queue.cc:260] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de61f9efed4348729d76fc670fded41d" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 44659 } }
I20250905 13:59:38.892513 27429 raft_consensus.cc:397] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:59:38.892773 27429 raft_consensus.cc:491] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:59:38.893056 27429 raft_consensus.cc:3058] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:59:38.896845 27429 raft_consensus.cc:513] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de61f9efed4348729d76fc670fded41d" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 44659 } }
I20250905 13:59:38.897487 27429 leader_election.cc:304] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de61f9efed4348729d76fc670fded41d; no voters:
I20250905 13:59:38.898943 27429 leader_election.cc:290] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:59:38.899549 27434 raft_consensus.cc:2802] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:59:38.901450 27434 raft_consensus.cc:695] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [term 1 LEADER]: Becoming Leader. State: Replica: de61f9efed4348729d76fc670fded41d, State: Running, Role: LEADER
I20250905 13:59:38.902087 27434 consensus_queue.cc:237] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de61f9efed4348729d76fc670fded41d" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 44659 } }
I20250905 13:59:38.903103 27429 sys_catalog.cc:564] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:59:38.910408 27435 sys_catalog.cc:455] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de61f9efed4348729d76fc670fded41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de61f9efed4348729d76fc670fded41d" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 44659 } } }
I20250905 13:59:38.911137 27435 sys_catalog.cc:458] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [sys.catalog]: This master's current role is: LEADER
I20250905 13:59:38.913877 27436 sys_catalog.cc:455] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [sys.catalog]: SysCatalogTable state changed. Reason: New leader de61f9efed4348729d76fc670fded41d. Latest consensus state: current_term: 1 leader_uuid: "de61f9efed4348729d76fc670fded41d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de61f9efed4348729d76fc670fded41d" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 44659 } } }
I20250905 13:59:38.914506 27436 sys_catalog.cc:458] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d [sys.catalog]: This master's current role is: LEADER
I20250905 13:59:38.916213 27443 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:59:38.926823 27443 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:59:38.941735 27443 catalog_manager.cc:1349] Generated new cluster ID: 128b7b09e25c4870b47256cbfb360ec9
I20250905 13:59:38.941942 27443 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:59:38.953981 27443 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 13:59:38.955191 27443 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:59:38.967231 27443 catalog_manager.cc:5955] T 00000000000000000000000000000000 P de61f9efed4348729d76fc670fded41d: Generated new TSK 0
I20250905 13:59:38.967958 27443 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 13:59:38.984932 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:0
--local_ip_for_outbound_sockets=127.21.98.193
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:44659
--builtin_ntp_servers=127.21.98.212:37663
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20250905 13:59:39.279157 27453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:39.279691 27453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:39.280439 27453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:39.310433 27453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:39.311208 27453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 13:59:39.341370 27453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:37663
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:44659
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:39.342607 27453 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:39.344310 27453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:39.356068 27459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:40.760012 27458 debug-util.cc:398] Leaking SignalData structure 0x7b0800006fc0 after lost signal to thread 27453
W20250905 13:59:41.125939 27458 kernel_stack_watchdog.cc:198] Thread 27453 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 400ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 13:59:39.357502 27460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:41.127518 27453 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.770s user 0.001s sys 0.001s
W20250905 13:59:41.127892 27453 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.770s user 0.001s sys 0.001s
W20250905 13:59:41.131108 27461 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1770 milliseconds
W20250905 13:59:41.132190 27462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:59:41.132288 27453 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:41.133770 27453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:41.136188 27453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:41.137683 27453 hybrid_clock.cc:648] HybridClock initialized: now 1757080781137633 us; error 40 us; skew 500 ppm
I20250905 13:59:41.138650 27453 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:41.145452 27453 webserver.cc:480] Webserver started at http://127.21.98.193:34155/ using document root <none> and password file <none>
I20250905 13:59:41.146565 27453 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:41.146808 27453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:41.147313 27453 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:41.153071 27453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "268aa6102a47471ba9e183beb2ac6274"
format_stamp: "Formatted at 2025-09-05 13:59:41 on dist-test-slave-g001"
I20250905 13:59:41.154457 27453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "268aa6102a47471ba9e183beb2ac6274"
format_stamp: "Formatted at 2025-09-05 13:59:41 on dist-test-slave-g001"
I20250905 13:59:41.162848 27453 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.005s
I20250905 13:59:41.169498 27469 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:41.170504 27453 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20250905 13:59:41.170837 27453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "268aa6102a47471ba9e183beb2ac6274"
format_stamp: "Formatted at 2025-09-05 13:59:41 on dist-test-slave-g001"
I20250905 13:59:41.171211 27453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:41.234477 27453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:41.236444 27453 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:41.236994 27453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:41.240167 27453 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:59:41.245604 27453 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:59:41.245850 27453 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:41.246134 27453 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:59:41.246313 27453 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:41.391425 27453 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:37477
I20250905 13:59:41.391486 27581 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:37477 every 8 connection(s)
I20250905 13:59:41.394841 27453 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20250905 13:59:41.403147 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 27453
I20250905 13:59:41.403885 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.ListTableCliSimpleParamTest.TestListTables_2.1757080637411871-21899-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20250905 13:59:41.428423 27582 heartbeater.cc:344] Connected to a master server at 127.21.98.254:44659
I20250905 13:59:41.428787 27582 heartbeater.cc:461] Registering TS with master...
I20250905 13:59:41.429706 27582 heartbeater.cc:507] Master 127.21.98.254:44659 requested a full tablet report, sending...
I20250905 13:59:41.432056 27394 ts_manager.cc:194] Registered new tserver with Master: 268aa6102a47471ba9e183beb2ac6274 (127.21.98.193:37477)
I20250905 13:59:41.434544 27394 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:41931
I20250905 13:59:41.439448 21899 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250905 13:59:41.475255 27394 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47052:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20250905 13:59:41.546097 27517 tablet_service.cc:1468] Processing CreateTablet for tablet ba6f3a83ea1a4dc0b39175ebc34e6641 (DEFAULT_TABLE table=TestTable [id=056f1f0f590e4693a94dee941d6e2e1d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:59:41.547777 27517 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba6f3a83ea1a4dc0b39175ebc34e6641. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:59:41.567620 27597 tablet_bootstrap.cc:492] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274: Bootstrap starting.
I20250905 13:59:41.574488 27597 tablet_bootstrap.cc:654] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274: Neither blocks nor log segments found. Creating new log.
I20250905 13:59:41.576758 27597 log.cc:826] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274: Log is configured to *not* fsync() on all Append() calls
I20250905 13:59:41.581709 27597 tablet_bootstrap.cc:492] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274: No bootstrap required, opened a new log
I20250905 13:59:41.582145 27597 ts_tablet_manager.cc:1397] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20250905 13:59:41.606983 27597 raft_consensus.cc:357] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268aa6102a47471ba9e183beb2ac6274" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37477 } }
I20250905 13:59:41.607630 27597 raft_consensus.cc:383] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:59:41.607885 27597 raft_consensus.cc:738] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268aa6102a47471ba9e183beb2ac6274, State: Initialized, Role: FOLLOWER
I20250905 13:59:41.608580 27597 consensus_queue.cc:260] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268aa6102a47471ba9e183beb2ac6274" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37477 } }
I20250905 13:59:41.609207 27597 raft_consensus.cc:397] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:59:41.609514 27597 raft_consensus.cc:491] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:59:41.609865 27597 raft_consensus.cc:3058] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:59:41.615784 27597 raft_consensus.cc:513] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268aa6102a47471ba9e183beb2ac6274" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37477 } }
I20250905 13:59:41.616643 27597 leader_election.cc:304] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 268aa6102a47471ba9e183beb2ac6274; no voters:
I20250905 13:59:41.618674 27597 leader_election.cc:290] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:59:41.619027 27599 raft_consensus.cc:2802] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:59:41.622165 27582 heartbeater.cc:499] Master 127.21.98.254:44659 was elected leader, sending a full tablet report...
I20250905 13:59:41.622857 27597 ts_tablet_manager.cc:1428] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274: Time spent starting tablet: real 0.040s user 0.033s sys 0.009s
I20250905 13:59:41.627481 27599 raft_consensus.cc:695] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [term 1 LEADER]: Becoming Leader. State: Replica: 268aa6102a47471ba9e183beb2ac6274, State: Running, Role: LEADER
I20250905 13:59:41.628201 27599 consensus_queue.cc:237] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268aa6102a47471ba9e183beb2ac6274" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37477 } }
I20250905 13:59:41.642222 27394 catalog_manager.cc:5582] T ba6f3a83ea1a4dc0b39175ebc34e6641 P 268aa6102a47471ba9e183beb2ac6274 reported cstate change: term changed from 0 to 1, leader changed from <none> to 268aa6102a47471ba9e183beb2ac6274 (127.21.98.193). New cstate: current_term: 1 leader_uuid: "268aa6102a47471ba9e183beb2ac6274" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "268aa6102a47471ba9e183beb2ac6274" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37477 } health_report { overall_health: HEALTHY } } }
I20250905 13:59:41.668125 21899 external_mini_cluster.cc:949] 1 TS(s) registered with all masters
I20250905 13:59:41.670946 21899 ts_itest-base.cc:246] Waiting for 1 tablets on tserver 268aa6102a47471ba9e183beb2ac6274 to finish bootstrapping
I20250905 13:59:44.242499 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 27453
I20250905 13:59:44.267879 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 27360
2025-09-05T13:59:44Z chronyd exiting
[ OK ] ListTableCliSimpleParamTest.TestListTables/2 (7162 ms)
[----------] 1 test from ListTableCliSimpleParamTest (7162 ms total)
[----------] 1 test from ListTableCliParamTest
[ RUN ] ListTableCliParamTest.ListTabletWithPartitionInfo/4
I20250905 13:59:44.322078 21899 test_util.cc:276] Using random seed: 1073593860
[ OK ] ListTableCliParamTest.ListTabletWithPartitionInfo/4 (11 ms)
[----------] 1 test from ListTableCliParamTest (11 ms total)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest
[ RUN ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0
2025-09-05T13:59:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T13:59:44Z Disabled control of system clock
I20250905 13:59:44.371143 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:36033
--webserver_interface=127.21.98.254
--webserver_port=0
--builtin_ntp_servers=127.21.98.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:36033 with env {}
W20250905 13:59:44.659300 27625 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:44.659761 27625 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:44.660163 27625 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:44.688330 27625 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 13:59:44.688611 27625 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:44.688784 27625 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 13:59:44.688947 27625 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 13:59:44.721802 27625 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:36033
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:36033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:44.723078 27625 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:44.724607 27625 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:44.735497 27631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:46.139286 27630 debug-util.cc:398] Leaking SignalData structure 0x7b0800037cc0 after lost signal to thread 27625
W20250905 13:59:46.439313 27630 kernel_stack_watchdog.cc:198] Thread 27625 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 401ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20250905 13:59:44.736048 27632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:46.439883 27625 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.705s user 0.000s sys 0.002s
W20250905 13:59:46.440217 27625 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.705s user 0.000s sys 0.003s
W20250905 13:59:46.440485 27633 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1703 milliseconds
W20250905 13:59:46.441708 27634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:59:46.441653 27625 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:46.444825 27625 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:46.447072 27625 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:46.448391 27625 hybrid_clock.cc:648] HybridClock initialized: now 1757080786448345 us; error 45 us; skew 500 ppm
I20250905 13:59:46.449110 27625 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:46.454923 27625 webserver.cc:480] Webserver started at http://127.21.98.254:45487/ using document root <none> and password file <none>
I20250905 13:59:46.455847 27625 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:46.456049 27625 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:46.456481 27625 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:46.460883 27625 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/instance:
uuid: "6d98a0e7b7504655a3fdf6cb7dd40832"
format_stamp: "Formatted at 2025-09-05 13:59:46 on dist-test-slave-g001"
I20250905 13:59:46.461957 27625 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal/instance:
uuid: "6d98a0e7b7504655a3fdf6cb7dd40832"
format_stamp: "Formatted at 2025-09-05 13:59:46 on dist-test-slave-g001"
I20250905 13:59:46.469192 27625 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20250905 13:59:46.474195 27641 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:46.475111 27625 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 13:59:46.475371 27625 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
uuid: "6d98a0e7b7504655a3fdf6cb7dd40832"
format_stamp: "Formatted at 2025-09-05 13:59:46 on dist-test-slave-g001"
I20250905 13:59:46.475628 27625 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:46.537006 27625 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:46.538486 27625 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:46.538944 27625 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:46.606319 27625 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:36033
I20250905 13:59:46.606379 27692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:36033 every 8 connection(s)
I20250905 13:59:46.608891 27625 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/info.pb
I20250905 13:59:46.614797 27693 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:59:46.616580 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 27625
I20250905 13:59:46.617139 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal/instance
I20250905 13:59:46.634550 27693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832: Bootstrap starting.
I20250905 13:59:46.639472 27693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832: Neither blocks nor log segments found. Creating new log.
I20250905 13:59:46.641040 27693 log.cc:826] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832: Log is configured to *not* fsync() on all Append() calls
I20250905 13:59:46.645685 27693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832: No bootstrap required, opened a new log
I20250905 13:59:46.661302 27693 raft_consensus.cc:357] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d98a0e7b7504655a3fdf6cb7dd40832" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } }
I20250905 13:59:46.661831 27693 raft_consensus.cc:383] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:59:46.662000 27693 raft_consensus.cc:738] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d98a0e7b7504655a3fdf6cb7dd40832, State: Initialized, Role: FOLLOWER
I20250905 13:59:46.662561 27693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d98a0e7b7504655a3fdf6cb7dd40832" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } }
I20250905 13:59:46.663075 27693 raft_consensus.cc:397] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 13:59:46.663309 27693 raft_consensus.cc:491] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 13:59:46.663698 27693 raft_consensus.cc:3058] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:59:46.667438 27693 raft_consensus.cc:513] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d98a0e7b7504655a3fdf6cb7dd40832" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } }
I20250905 13:59:46.667989 27693 leader_election.cc:304] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6d98a0e7b7504655a3fdf6cb7dd40832; no voters:
I20250905 13:59:46.669433 27693 leader_election.cc:290] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 13:59:46.669998 27698 raft_consensus.cc:2802] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:59:46.672168 27698 raft_consensus.cc:695] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [term 1 LEADER]: Becoming Leader. State: Replica: 6d98a0e7b7504655a3fdf6cb7dd40832, State: Running, Role: LEADER
I20250905 13:59:46.672981 27698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d98a0e7b7504655a3fdf6cb7dd40832" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } }
I20250905 13:59:46.674373 27693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 13:59:46.683110 27700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6d98a0e7b7504655a3fdf6cb7dd40832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d98a0e7b7504655a3fdf6cb7dd40832" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } } }
I20250905 13:59:46.683192 27699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6d98a0e7b7504655a3fdf6cb7dd40832. Latest consensus state: current_term: 1 leader_uuid: "6d98a0e7b7504655a3fdf6cb7dd40832" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6d98a0e7b7504655a3fdf6cb7dd40832" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } } }
I20250905 13:59:46.684077 27699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [sys.catalog]: This master's current role is: LEADER
I20250905 13:59:46.684044 27700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832 [sys.catalog]: This master's current role is: LEADER
I20250905 13:59:46.687239 27706 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 13:59:46.696987 27706 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 13:59:46.712615 27706 catalog_manager.cc:1349] Generated new cluster ID: 8784f6b606ac4deb83b53ffcc2b57c8e
I20250905 13:59:46.712905 27706 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 13:59:46.745798 27706 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 13:59:46.747696 27706 catalog_manager.cc:1506] Loading token signing keys...
I20250905 13:59:46.765555 27706 catalog_manager.cc:5955] T 00000000000000000000000000000000 P 6d98a0e7b7504655a3fdf6cb7dd40832: Generated new TSK 0
I20250905 13:59:46.766376 27706 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 13:59:46.778805 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:0
--local_ip_for_outbound_sockets=127.21.98.193
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36033
--builtin_ntp_servers=127.21.98.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20250905 13:59:47.085337 27717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:47.085837 27717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:47.086287 27717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:47.115839 27717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:47.116765 27717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 13:59:47.149163 27717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:47.150386 27717 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:47.152027 27717 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:47.163829 27723 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:48.568156 27722 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 27717
W20250905 13:59:48.889724 27725 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1724 milliseconds
W20250905 13:59:48.888022 27717 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.723s user 0.659s sys 1.041s
W20250905 13:59:48.890275 27726 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:48.890250 27717 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.725s user 0.659s sys 1.041s
I20250905 13:59:48.891110 27717 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250905 13:59:47.165652 27724 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:59:48.895118 27717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:48.897635 27717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:48.899093 27717 hybrid_clock.cc:648] HybridClock initialized: now 1757080788899049 us; error 35 us; skew 500 ppm
I20250905 13:59:48.900156 27717 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:48.907418 27717 webserver.cc:480] Webserver started at http://127.21.98.193:39429/ using document root <none> and password file <none>
I20250905 13:59:48.908628 27717 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:48.908888 27717 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:48.909395 27717 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:48.916894 27717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/instance:
uuid: "e4bef33a2dbd44768749b509e0d53995"
format_stamp: "Formatted at 2025-09-05 13:59:48 on dist-test-slave-g001"
I20250905 13:59:48.918318 27717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal/instance:
uuid: "e4bef33a2dbd44768749b509e0d53995"
format_stamp: "Formatted at 2025-09-05 13:59:48 on dist-test-slave-g001"
I20250905 13:59:48.927407 27717 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.002s sys 0.006s
I20250905 13:59:48.934106 27734 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:48.935194 27717 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250905 13:59:48.935614 27717 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal
uuid: "e4bef33a2dbd44768749b509e0d53995"
format_stamp: "Formatted at 2025-09-05 13:59:48 on dist-test-slave-g001"
I20250905 13:59:48.936013 27717 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:48.991494 27717 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:48.992910 27717 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:48.993337 27717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:48.995791 27717 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:59:49.000150 27717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:59:49.000335 27717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:49.000564 27717 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:59:49.000715 27717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:49.164960 27717 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:37031
I20250905 13:59:49.165123 27846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:37031 every 8 connection(s)
I20250905 13:59:49.167521 27717 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/info.pb
I20250905 13:59:49.176662 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 27717
I20250905 13:59:49.177225 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal/instance
I20250905 13:59:49.183904 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.194:0
--local_ip_for_outbound_sockets=127.21.98.194
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36033
--builtin_ntp_servers=127.21.98.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250905 13:59:49.192476 27847 heartbeater.cc:344] Connected to a master server at 127.21.98.254:36033
I20250905 13:59:49.192961 27847 heartbeater.cc:461] Registering TS with master...
I20250905 13:59:49.194239 27847 heartbeater.cc:507] Master 127.21.98.254:36033 requested a full tablet report, sending...
I20250905 13:59:49.197253 27658 ts_manager.cc:194] Registered new tserver with Master: e4bef33a2dbd44768749b509e0d53995 (127.21.98.193:37031)
I20250905 13:59:49.200122 27658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:44371
W20250905 13:59:49.475847 27851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:49.476315 27851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:49.476801 27851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:49.504947 27851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:49.505590 27851 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.194
I20250905 13:59:49.536423 27851 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.98.194
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.194
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:49.537544 27851 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:49.539011 27851 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:49.548954 27857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 13:59:50.203950 27847 heartbeater.cc:499] Master 127.21.98.254:36033 was elected leader, sending a full tablet report...
W20250905 13:59:49.550398 27858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:50.653447 27860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:50.655812 27859 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1101 milliseconds
I20250905 13:59:50.655975 27851 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:50.657284 27851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:50.659269 27851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:50.660612 27851 hybrid_clock.cc:648] HybridClock initialized: now 1757080790660550 us; error 41 us; skew 500 ppm
I20250905 13:59:50.661336 27851 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:50.668074 27851 webserver.cc:480] Webserver started at http://127.21.98.194:43529/ using document root <none> and password file <none>
I20250905 13:59:50.668897 27851 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:50.669083 27851 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:50.669499 27851 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:50.673658 27851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/instance:
uuid: "f62993cab8bf4bc7b34b11fae92969a2"
format_stamp: "Formatted at 2025-09-05 13:59:50 on dist-test-slave-g001"
I20250905 13:59:50.674778 27851 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal/instance:
uuid: "f62993cab8bf4bc7b34b11fae92969a2"
format_stamp: "Formatted at 2025-09-05 13:59:50 on dist-test-slave-g001"
I20250905 13:59:50.681710 27851 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20250905 13:59:50.687048 27867 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:50.687953 27851 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 13:59:50.688268 27851 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal
uuid: "f62993cab8bf4bc7b34b11fae92969a2"
format_stamp: "Formatted at 2025-09-05 13:59:50 on dist-test-slave-g001"
I20250905 13:59:50.688575 27851 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:50.734639 27851 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:50.736008 27851 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:50.736397 27851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:50.738847 27851 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:59:50.742444 27851 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:59:50.742736 27851 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:50.743027 27851 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:59:50.743249 27851 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:50.890391 27851 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.194:41267
I20250905 13:59:50.890487 27979 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.194:41267 every 8 connection(s)
I20250905 13:59:50.893054 27851 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/info.pb
I20250905 13:59:50.902855 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 27851
I20250905 13:59:50.903271 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal/instance
I20250905 13:59:50.909448 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.195:0
--local_ip_for_outbound_sockets=127.21.98.195
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36033
--builtin_ntp_servers=127.21.98.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250905 13:59:50.913091 27980 heartbeater.cc:344] Connected to a master server at 127.21.98.254:36033
I20250905 13:59:50.913462 27980 heartbeater.cc:461] Registering TS with master...
I20250905 13:59:50.914408 27980 heartbeater.cc:507] Master 127.21.98.254:36033 requested a full tablet report, sending...
I20250905 13:59:50.916381 27658 ts_manager.cc:194] Registered new tserver with Master: f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194:41267)
I20250905 13:59:50.918058 27658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.194:58253
W20250905 13:59:51.207620 27984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:51.208312 27984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 13:59:51.209132 27984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:51.258244 27984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 13:59:51.259042 27984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.195
I20250905 13:59:51.291993 27984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.98.195
--webserver_port=0
--tserver_master_addrs=127.21.98.254:36033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.195
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 13:59:51.293247 27984 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 13:59:51.294801 27984 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 13:59:51.307385 27990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:51.771214 27689 debug-util.cc:398] Leaking SignalData structure 0x7b080006f060 after lost signal to thread 27626
I20250905 13:59:51.921203 27980 heartbeater.cc:499] Master 127.21.98.254:36033 was elected leader, sending a full tablet report...
W20250905 13:59:52.248183 27976 debug-util.cc:398] Leaking SignalData structure 0x7b08000ac060 after lost signal to thread 27852
W20250905 13:59:52.249583 27976 debug-util.cc:398] Leaking SignalData structure 0x7b08000acf40 after lost signal to thread 27979
W20250905 13:59:51.307579 27991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:52.546263 27993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 13:59:52.548481 27992 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1240 milliseconds
I20250905 13:59:52.548650 27984 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 13:59:52.550187 27984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 13:59:52.555363 27984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 13:59:52.556820 27984 hybrid_clock.cc:648] HybridClock initialized: now 1757080792556748 us; error 67 us; skew 500 ppm
I20250905 13:59:52.558014 27984 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 13:59:52.567499 27984 webserver.cc:480] Webserver started at http://127.21.98.195:45537/ using document root <none> and password file <none>
I20250905 13:59:52.568755 27984 fs_manager.cc:362] Metadata directory not provided
I20250905 13:59:52.569051 27984 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 13:59:52.569670 27984 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 13:59:52.576532 27984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/instance:
uuid: "62fc44fafab2486e99bd94370b83619e"
format_stamp: "Formatted at 2025-09-05 13:59:52 on dist-test-slave-g001"
I20250905 13:59:52.578258 27984 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal/instance:
uuid: "62fc44fafab2486e99bd94370b83619e"
format_stamp: "Formatted at 2025-09-05 13:59:52 on dist-test-slave-g001"
I20250905 13:59:52.587787 27984 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.000s
I20250905 13:59:52.595919 28000 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:52.597198 27984 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20250905 13:59:52.597724 27984 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal
uuid: "62fc44fafab2486e99bd94370b83619e"
format_stamp: "Formatted at 2025-09-05 13:59:52 on dist-test-slave-g001"
I20250905 13:59:52.598260 27984 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 13:59:52.682698 27984 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 13:59:52.684799 27984 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 13:59:52.685350 27984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 13:59:52.688565 27984 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 13:59:52.693908 27984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 13:59:52.694265 27984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20250905 13:59:52.694641 27984 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 13:59:52.694975 27984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 13:59:52.862684 27984 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.195:44685
I20250905 13:59:52.862813 28112 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.195:44685 every 8 connection(s)
I20250905 13:59:52.866473 27984 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/info.pb
I20250905 13:59:52.869891 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 27984
I20250905 13:59:52.870332 21899 external_mini_cluster.cc:1442] Reading /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal/instance
I20250905 13:59:52.924191 28113 heartbeater.cc:344] Connected to a master server at 127.21.98.254:36033
I20250905 13:59:52.924695 28113 heartbeater.cc:461] Registering TS with master...
I20250905 13:59:52.926064 28113 heartbeater.cc:507] Master 127.21.98.254:36033 requested a full tablet report, sending...
I20250905 13:59:52.928653 27658 ts_manager.cc:194] Registered new tserver with Master: 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685)
I20250905 13:59:52.930321 27658 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.195:60747
I20250905 13:59:52.935654 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 13:59:52.978478 21899 test_util.cc:276] Using random seed: 1082250260
I20250905 13:59:53.038626 27658 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37568:
name: "pre_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 13:59:53.042187 27658 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table pre_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 13:59:53.123631 27782 tablet_service.cc:1468] Processing CreateTablet for tablet 5a6e84b9ab724881a41dc83db6bafe95 (DEFAULT_TABLE table=pre_rebuild [id=d601deabb3344392a826fa0d93be7ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:59:53.125759 27782 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a6e84b9ab724881a41dc83db6bafe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:59:53.131646 28048 tablet_service.cc:1468] Processing CreateTablet for tablet 5a6e84b9ab724881a41dc83db6bafe95 (DEFAULT_TABLE table=pre_rebuild [id=d601deabb3344392a826fa0d93be7ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:59:53.133634 28048 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a6e84b9ab724881a41dc83db6bafe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:59:53.142426 27915 tablet_service.cc:1468] Processing CreateTablet for tablet 5a6e84b9ab724881a41dc83db6bafe95 (DEFAULT_TABLE table=pre_rebuild [id=d601deabb3344392a826fa0d93be7ddb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 13:59:53.144232 27915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a6e84b9ab724881a41dc83db6bafe95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 13:59:53.167614 28137 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Bootstrap starting.
I20250905 13:59:53.177357 28137 tablet_bootstrap.cc:654] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Neither blocks nor log segments found. Creating new log.
I20250905 13:59:53.179757 28137 log.cc:826] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Log is configured to *not* fsync() on all Append() calls
I20250905 13:59:53.190618 28137 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: No bootstrap required, opened a new log
I20250905 13:59:53.191216 28137 ts_tablet_manager.cc:1397] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Time spent bootstrapping tablet: real 0.024s user 0.017s sys 0.003s
I20250905 13:59:53.200035 28139 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Bootstrap starting.
I20250905 13:59:53.216557 28140 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Bootstrap starting.
I20250905 13:59:53.219641 28137 raft_consensus.cc:357] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 13:59:53.220754 28137 raft_consensus.cc:383] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:59:53.221153 28137 raft_consensus.cc:738] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fc44fafab2486e99bd94370b83619e, State: Initialized, Role: FOLLOWER
I20250905 13:59:53.222671 28137 consensus_queue.cc:260] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 13:59:53.223941 28139 tablet_bootstrap.cc:654] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Neither blocks nor log segments found. Creating new log.
I20250905 13:59:53.226385 28139 log.cc:826] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Log is configured to *not* fsync() on all Append() calls
I20250905 13:59:53.231585 28113 heartbeater.cc:499] Master 127.21.98.254:36033 was elected leader, sending a full tablet report...
I20250905 13:59:53.233047 28137 ts_tablet_manager.cc:1428] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Time spent starting tablet: real 0.041s user 0.032s sys 0.004s
I20250905 13:59:53.240825 28139 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: No bootstrap required, opened a new log
I20250905 13:59:53.241356 28139 ts_tablet_manager.cc:1397] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Time spent bootstrapping tablet: real 0.042s user 0.015s sys 0.002s
I20250905 13:59:53.243703 28140 tablet_bootstrap.cc:654] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Neither blocks nor log segments found. Creating new log.
I20250905 13:59:53.246225 28140 log.cc:826] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Log is configured to *not* fsync() on all Append() calls
I20250905 13:59:53.253121 28141 raft_consensus.cc:491] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:59:53.253793 28141 raft_consensus.cc:513] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 13:59:53.261797 28140 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: No bootstrap required, opened a new log
I20250905 13:59:53.262363 28140 ts_tablet_manager.cc:1397] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Time spent bootstrapping tablet: real 0.046s user 0.012s sys 0.006s
I20250905 13:59:53.265805 28141 leader_election.cc:290] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4bef33a2dbd44768749b509e0d53995 (127.21.98.193:37031), f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194:41267)
I20250905 13:59:53.285301 27935 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a6e84b9ab724881a41dc83db6bafe95" candidate_uuid: "62fc44fafab2486e99bd94370b83619e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62993cab8bf4bc7b34b11fae92969a2" is_pre_election: true
I20250905 13:59:53.286713 27802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a6e84b9ab724881a41dc83db6bafe95" candidate_uuid: "62fc44fafab2486e99bd94370b83619e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bef33a2dbd44768749b509e0d53995" is_pre_election: true
W20250905 13:59:53.288770 28003 leader_election.cc:343] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194:41267): Illegal state: must be running to vote when last-logged opid is not known
I20250905 13:59:53.288429 28139 raft_consensus.cc:357] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
W20250905 13:59:53.289559 28003 leader_election.cc:343] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e4bef33a2dbd44768749b509e0d53995 (127.21.98.193:37031): Illegal state: must be running to vote when last-logged opid is not known
I20250905 13:59:53.289599 28139 raft_consensus.cc:383] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:59:53.289942 28139 raft_consensus.cc:738] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bef33a2dbd44768749b509e0d53995, State: Initialized, Role: FOLLOWER
I20250905 13:59:53.290055 28003 leader_election.cc:304] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 62fc44fafab2486e99bd94370b83619e; no voters: e4bef33a2dbd44768749b509e0d53995, f62993cab8bf4bc7b34b11fae92969a2
I20250905 13:59:53.290982 28141 raft_consensus.cc:2747] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 13:59:53.291016 28139 consensus_queue.cc:260] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 13:59:53.295311 28139 ts_tablet_manager.cc:1428] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Time spent starting tablet: real 0.054s user 0.025s sys 0.007s
I20250905 13:59:53.305620 28140 raft_consensus.cc:357] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 13:59:53.306545 28140 raft_consensus.cc:383] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 13:59:53.306918 28140 raft_consensus.cc:738] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62993cab8bf4bc7b34b11fae92969a2, State: Initialized, Role: FOLLOWER
I20250905 13:59:53.307865 28140 consensus_queue.cc:260] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 13:59:53.311933 28140 ts_tablet_manager.cc:1428] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Time spent starting tablet: real 0.049s user 0.032s sys 0.003s
I20250905 13:59:53.336277 28146 raft_consensus.cc:491] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 13:59:53.336935 28146 raft_consensus.cc:513] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 13:59:53.340026 28146 leader_election.cc:290] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4bef33a2dbd44768749b509e0d53995 (127.21.98.193:37031), 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685)
I20250905 13:59:53.364995 27802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a6e84b9ab724881a41dc83db6bafe95" candidate_uuid: "f62993cab8bf4bc7b34b11fae92969a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bef33a2dbd44768749b509e0d53995" is_pre_election: true
I20250905 13:59:53.365694 27802 raft_consensus.cc:2466] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f62993cab8bf4bc7b34b11fae92969a2 in term 0.
I20250905 13:59:53.367203 27870 leader_election.cc:304] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4bef33a2dbd44768749b509e0d53995, f62993cab8bf4bc7b34b11fae92969a2; no voters:
I20250905 13:59:53.368155 28146 raft_consensus.cc:2802] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 13:59:53.368496 28146 raft_consensus.cc:491] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 13:59:53.368834 28146 raft_consensus.cc:3058] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:59:53.370437 28068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a6e84b9ab724881a41dc83db6bafe95" candidate_uuid: "f62993cab8bf4bc7b34b11fae92969a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fc44fafab2486e99bd94370b83619e" is_pre_election: true
I20250905 13:59:53.371248 28068 raft_consensus.cc:2466] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f62993cab8bf4bc7b34b11fae92969a2 in term 0.
I20250905 13:59:53.375226 28146 raft_consensus.cc:513] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 13:59:53.377847 27802 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a6e84b9ab724881a41dc83db6bafe95" candidate_uuid: "f62993cab8bf4bc7b34b11fae92969a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4bef33a2dbd44768749b509e0d53995"
I20250905 13:59:53.378001 28068 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a6e84b9ab724881a41dc83db6bafe95" candidate_uuid: "f62993cab8bf4bc7b34b11fae92969a2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fc44fafab2486e99bd94370b83619e"
I20250905 13:59:53.378324 27802 raft_consensus.cc:3058] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:59:53.378420 28068 raft_consensus.cc:3058] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Advancing to term 1
I20250905 13:59:53.384902 27802 raft_consensus.cc:2466] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f62993cab8bf4bc7b34b11fae92969a2 in term 1.
I20250905 13:59:53.384910 28068 raft_consensus.cc:2466] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f62993cab8bf4bc7b34b11fae92969a2 in term 1.
I20250905 13:59:53.385891 27868 leader_election.cc:304] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62fc44fafab2486e99bd94370b83619e, f62993cab8bf4bc7b34b11fae92969a2; no voters:
I20250905 13:59:53.388175 28146 leader_election.cc:290] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [CANDIDATE]: Term 1 election: Requested vote from peers e4bef33a2dbd44768749b509e0d53995 (127.21.98.193:37031), 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685)
I20250905 13:59:53.388468 28149 raft_consensus.cc:2802] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 13:59:53.389348 28149 raft_consensus.cc:695] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 1 LEADER]: Becoming Leader. State: Replica: f62993cab8bf4bc7b34b11fae92969a2, State: Running, Role: LEADER
I20250905 13:59:53.390244 28149 consensus_queue.cc:237] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
W20250905 13:59:53.392364 28116 tablet.cc:2378] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20250905 13:59:53.399933 27981 tablet.cc:2378] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:59:53.405867 27658 catalog_manager.cc:5582] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 reported cstate change: term changed from 0 to 1, leader changed from <none> to f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194). New cstate: current_term: 1 leader_uuid: "f62993cab8bf4bc7b34b11fae92969a2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } health_report { overall_health: UNKNOWN } } }
W20250905 13:59:53.426262 27848 tablet.cc:2378] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 13:59:53.675055 28068 raft_consensus.cc:1273] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 1 FOLLOWER]: Refusing update from remote peer f62993cab8bf4bc7b34b11fae92969a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 13:59:53.675104 27802 raft_consensus.cc:1273] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Refusing update from remote peer f62993cab8bf4bc7b34b11fae92969a2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 13:59:53.676993 28149 consensus_queue.cc:1035] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 13:59:53.678562 28146 consensus_queue.cc:1035] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 13:59:53.727849 28158 mvcc.cc:204] Tried to move back new op lower bound from 7197002930875117568 to 7197002929735950336. Current Snapshot: MvccSnapshot[applied={T|T < 7197002930875117568}]
I20250905 13:59:53.757053 28157 mvcc.cc:204] Tried to move back new op lower bound from 7197002930875117568 to 7197002929735950336. Current Snapshot: MvccSnapshot[applied={T|T < 7197002930875117568}]
I20250905 13:59:53.764518 28159 mvcc.cc:204] Tried to move back new op lower bound from 7197002930875117568 to 7197002929735950336. Current Snapshot: MvccSnapshot[applied={T|T < 7197002930875117568}]
I20250905 13:59:58.786795 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 27625
W20250905 13:59:58.828789 27980 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:36033 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:36033: connect: Connection refused (error 111)
W20250905 13:59:58.848171 27847 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:36033 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:36033: connect: Connection refused (error 111)
W20250905 13:59:59.147214 28193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 13:59:59.147744 28193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 13:59:59.179400 28193 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
W20250905 13:59:59.798157 28113 heartbeater.cc:646] Failed to heartbeat to 127.21.98.254:36033 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.21.98.254:36033: connect: Connection refused (error 111)
W20250905 14:00:00.589847 28193 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 1.367s user 0.532s sys 0.831s
W20250905 14:00:00.590204 28193 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 1.367s user 0.532s sys 0.831s
I20250905 14:00:00.722205 28193 minidump.cc:252] Setting minidump size limit to 20M
I20250905 14:00:00.724542 28193 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 14:00:00.725672 28193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 14:00:00.735508 28225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 14:00:00.735616 28226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 14:00:00.911340 28228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 14:00:00.911528 28193 server_base.cc:1047] running on GCE node
I20250905 14:00:00.913043 28193 hybrid_clock.cc:584] initializing the hybrid clock with 'system' time source
I20250905 14:00:00.913628 28193 hybrid_clock.cc:648] HybridClock initialized: now 1757080800913602 us; error 60749 us; skew 500 ppm
I20250905 14:00:00.914614 28193 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 14:00:00.921198 28193 webserver.cc:480] Webserver started at http://0.0.0.0:33667/ using document root <none> and password file <none>
I20250905 14:00:00.922425 28193 fs_manager.cc:362] Metadata directory not provided
I20250905 14:00:00.922721 28193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 14:00:00.923292 28193 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 14:00:00.929864 28193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/instance:
uuid: "a53cd829c39a47bfaba2917be9d0645c"
format_stamp: "Formatted at 2025-09-05 14:00:00 on dist-test-slave-g001"
I20250905 14:00:00.931358 28193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal/instance:
uuid: "a53cd829c39a47bfaba2917be9d0645c"
format_stamp: "Formatted at 2025-09-05 14:00:00 on dist-test-slave-g001"
I20250905 14:00:00.939481 28193 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20250905 14:00:00.946491 28236 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 14:00:00.947453 28193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 14:00:00.947837 28193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
uuid: "a53cd829c39a47bfaba2917be9d0645c"
format_stamp: "Formatted at 2025-09-05 14:00:00 on dist-test-slave-g001"
I20250905 14:00:00.948244 28193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 14:00:01.119192 28193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 14:00:01.120422 28193 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 14:00:01.120798 28193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 14:00:01.124949 28193 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 14:00:01.138098 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: Bootstrap starting.
I20250905 14:00:01.142416 28193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: Neither blocks nor log segments found. Creating new log.
I20250905 14:00:01.143854 28193 log.cc:826] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: Log is configured to *not* fsync() on all Append() calls
I20250905 14:00:01.147423 28193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: No bootstrap required, opened a new log
I20250905 14:00:01.161631 28193 raft_consensus.cc:357] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER }
I20250905 14:00:01.162026 28193 raft_consensus.cc:383] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 14:00:01.162247 28193 raft_consensus.cc:738] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53cd829c39a47bfaba2917be9d0645c, State: Initialized, Role: FOLLOWER
I20250905 14:00:01.163045 28193 consensus_queue.cc:260] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER }
I20250905 14:00:01.163502 28193 raft_consensus.cc:397] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 14:00:01.163743 28193 raft_consensus.cc:491] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 14:00:01.163998 28193 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 0 FOLLOWER]: Advancing to term 1
I20250905 14:00:01.167474 28193 raft_consensus.cc:513] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER }
I20250905 14:00:01.168061 28193 leader_election.cc:304] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a53cd829c39a47bfaba2917be9d0645c; no voters:
I20250905 14:00:01.169548 28193 leader_election.cc:290] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [CANDIDATE]: Term 1 election: Requested vote from peers
I20250905 14:00:01.169804 28243 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 1 FOLLOWER]: Leader election won for term 1
I20250905 14:00:01.171979 28243 raft_consensus.cc:695] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 1 LEADER]: Becoming Leader. State: Replica: a53cd829c39a47bfaba2917be9d0645c, State: Running, Role: LEADER
I20250905 14:00:01.172852 28243 consensus_queue.cc:237] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER }
I20250905 14:00:01.179714 28244 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a53cd829c39a47bfaba2917be9d0645c. Latest consensus state: current_term: 1 leader_uuid: "a53cd829c39a47bfaba2917be9d0645c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER } }
I20250905 14:00:01.180325 28244 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [sys.catalog]: This master's current role is: LEADER
I20250905 14:00:01.181957 28245 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a53cd829c39a47bfaba2917be9d0645c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER } }
I20250905 14:00:01.182554 28245 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [sys.catalog]: This master's current role is: LEADER
I20250905 14:00:01.193495 28193 tablet_replica.cc:331] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: stopping tablet replica
I20250905 14:00:01.194077 28193 raft_consensus.cc:2241] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 1 LEADER]: Raft consensus shutting down.
I20250905 14:00:01.194547 28193 raft_consensus.cc:2270] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 14:00:01.196810 28193 master.cc:561] Master@0.0.0.0:7051 shutting down...
W20250905 14:00:01.197330 28193 acceptor_pool.cc:196] Could not shut down acceptor socket on 0.0.0.0:7051: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20250905 14:00:01.224717 28193 master.cc:583] Master@0.0.0.0:7051 shutdown complete.
I20250905 14:00:02.253957 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 27717
I20250905 14:00:02.289178 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 27851
I20250905 14:00:02.328334 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 27984
I20250905 14:00:02.359993 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:36033
--webserver_interface=127.21.98.254
--webserver_port=45487
--builtin_ntp_servers=127.21.98.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.21.98.254:36033 with env {}
W20250905 14:00:02.638187 28253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 14:00:02.638684 28253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 14:00:02.639153 28253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 14:00:02.666087 28253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 14:00:02.666339 28253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 14:00:02.666617 28253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 14:00:02.666875 28253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 14:00:02.697167 28253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.21.98.254:36033
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.21.98.254:36033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.21.98.254
--webserver_port=45487
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 14:00:02.698434 28253 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 14:00:02.700027 28253 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 14:00:02.709431 28259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 14:00:02.711175 28260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 14:00:03.801458 28262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 14:00:03.803107 28261 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1089 milliseconds
I20250905 14:00:03.803193 28253 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 14:00:03.804369 28253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 14:00:03.806735 28253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 14:00:03.808215 28253 hybrid_clock.cc:648] HybridClock initialized: now 1757080803808185 us; error 34 us; skew 500 ppm
I20250905 14:00:03.808902 28253 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 14:00:03.814123 28253 webserver.cc:480] Webserver started at http://127.21.98.254:45487/ using document root <none> and password file <none>
I20250905 14:00:03.815033 28253 fs_manager.cc:362] Metadata directory not provided
I20250905 14:00:03.815267 28253 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 14:00:03.822283 28253 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20250905 14:00:03.826310 28269 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 14:00:03.827203 28253 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20250905 14:00:03.827481 28253 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
uuid: "a53cd829c39a47bfaba2917be9d0645c"
format_stamp: "Formatted at 2025-09-05 14:00:00 on dist-test-slave-g001"
I20250905 14:00:03.829098 28253 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 14:00:03.883728 28253 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 14:00:03.885152 28253 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 14:00:03.885567 28253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 14:00:03.950225 28253 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.254:36033
I20250905 14:00:03.950286 28320 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.254:36033 every 8 connection(s)
I20250905 14:00:03.952790 28253 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/info.pb
I20250905 14:00:03.962637 28321 sys_catalog.cc:263] Verifying existing consensus state
I20250905 14:00:03.963330 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 28253
I20250905 14:00:03.964609 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.193:37031
--local_ip_for_outbound_sockets=127.21.98.193
--tserver_master_addrs=127.21.98.254:36033
--webserver_port=39429
--webserver_interface=127.21.98.193
--builtin_ntp_servers=127.21.98.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250905 14:00:03.972687 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: Bootstrap starting.
I20250905 14:00:03.985267 28321 log.cc:826] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: Log is configured to *not* fsync() on all Append() calls
I20250905 14:00:03.999260 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=2 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 14:00:04.000059 28321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: Bootstrap complete.
I20250905 14:00:04.027338 28321 raft_consensus.cc:357] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } }
I20250905 14:00:04.028000 28321 raft_consensus.cc:738] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a53cd829c39a47bfaba2917be9d0645c, State: Initialized, Role: FOLLOWER
I20250905 14:00:04.028916 28321 consensus_queue.cc:260] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } }
I20250905 14:00:04.029484 28321 raft_consensus.cc:397] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20250905 14:00:04.029763 28321 raft_consensus.cc:491] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20250905 14:00:04.030157 28321 raft_consensus.cc:3058] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 1 FOLLOWER]: Advancing to term 2
I20250905 14:00:04.036160 28321 raft_consensus.cc:513] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } }
I20250905 14:00:04.036965 28321 leader_election.cc:304] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a53cd829c39a47bfaba2917be9d0645c; no voters:
I20250905 14:00:04.038894 28321 leader_election.cc:290] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [CANDIDATE]: Term 2 election: Requested vote from peers
I20250905 14:00:04.039244 28325 raft_consensus.cc:2802] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 2 FOLLOWER]: Leader election won for term 2
I20250905 14:00:04.041968 28325 raft_consensus.cc:695] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [term 2 LEADER]: Becoming Leader. State: Replica: a53cd829c39a47bfaba2917be9d0645c, State: Running, Role: LEADER
I20250905 14:00:04.042618 28325 consensus_queue.cc:237] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } }
I20250905 14:00:04.043012 28321 sys_catalog.cc:564] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [sys.catalog]: configured and running, proceeding with master startup.
I20250905 14:00:04.052009 28326 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a53cd829c39a47bfaba2917be9d0645c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } } }
I20250905 14:00:04.054232 28326 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [sys.catalog]: This master's current role is: LEADER
I20250905 14:00:04.053494 28327 sys_catalog.cc:455] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a53cd829c39a47bfaba2917be9d0645c. Latest consensus state: current_term: 2 leader_uuid: "a53cd829c39a47bfaba2917be9d0645c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a53cd829c39a47bfaba2917be9d0645c" member_type: VOTER last_known_addr { host: "127.21.98.254" port: 36033 } } }
I20250905 14:00:04.055735 28327 sys_catalog.cc:458] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c [sys.catalog]: This master's current role is: LEADER
I20250905 14:00:04.062404 28332 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 14:00:04.072654 28332 catalog_manager.cc:671] Loaded metadata for table pre_rebuild [id=1020fee07c1643b1bcda03bd22e574e6]
I20250905 14:00:04.078908 28332 tablet_loader.cc:96] loaded metadata for tablet 5a6e84b9ab724881a41dc83db6bafe95 (table pre_rebuild [id=1020fee07c1643b1bcda03bd22e574e6])
I20250905 14:00:04.080195 28332 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 14:00:04.103229 28332 catalog_manager.cc:1349] Generated new cluster ID: cd1fef8f69c941cf9779219ec25f4301
I20250905 14:00:04.103506 28332 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 14:00:04.120319 28332 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 14:00:04.121451 28332 catalog_manager.cc:1506] Loading token signing keys...
I20250905 14:00:04.130548 28332 catalog_manager.cc:5955] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: Generated new TSK 0
I20250905 14:00:04.131229 28332 catalog_manager.cc:1516] Initializing in-progress tserver states...
W20250905 14:00:04.314015 28323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 14:00:04.314482 28323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 14:00:04.315042 28323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 14:00:04.343705 28323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 14:00:04.344480 28323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.193
I20250905 14:00:04.375581 28323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.193:37031
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.21.98.193
--webserver_port=39429
--tserver_master_addrs=127.21.98.254:36033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.193
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 14:00:04.376821 28323 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 14:00:04.378345 28323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 14:00:04.390170 28349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 14:00:04.392673 28350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 14:00:04.393986 28352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 14:00:05.726478 28351 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1333 milliseconds
I20250905 14:00:05.726537 28323 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20250905 14:00:05.731340 28323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 14:00:05.733808 28323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 14:00:05.735251 28323 hybrid_clock.cc:648] HybridClock initialized: now 1757080805735210 us; error 60 us; skew 500 ppm
I20250905 14:00:05.736119 28323 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 14:00:05.742791 28323 webserver.cc:480] Webserver started at http://127.21.98.193:39429/ using document root <none> and password file <none>
I20250905 14:00:05.743656 28323 fs_manager.cc:362] Metadata directory not provided
I20250905 14:00:05.743893 28323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 14:00:05.751786 28323 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.008s sys 0.000s
I20250905 14:00:05.757890 28359 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 14:00:05.759284 28323 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20250905 14:00:05.759883 28323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal
uuid: "e4bef33a2dbd44768749b509e0d53995"
format_stamp: "Formatted at 2025-09-05 13:59:48 on dist-test-slave-g001"
I20250905 14:00:05.762784 28323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 14:00:05.847576 28323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 14:00:05.848937 28323 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 14:00:05.849357 28323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 14:00:05.852486 28323 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 14:00:05.859055 28366 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 14:00:05.866581 28323 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 14:00:05.866781 28323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.001s
I20250905 14:00:05.867122 28323 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 14:00:05.871155 28323 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 14:00:05.871382 28323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.004s sys 0.000s
I20250905 14:00:05.871786 28366 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Bootstrap starting.
I20250905 14:00:06.068809 28323 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.193:37031
I20250905 14:00:06.068970 28472 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.193:37031 every 8 connection(s)
I20250905 14:00:06.072624 28323 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/info.pb
I20250905 14:00:06.074101 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 28323
I20250905 14:00:06.075999 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.194:41267
--local_ip_for_outbound_sockets=127.21.98.194
--tserver_master_addrs=127.21.98.254:36033
--webserver_port=43529
--webserver_interface=127.21.98.194
--builtin_ntp_servers=127.21.98.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250905 14:00:06.131515 28473 heartbeater.cc:344] Connected to a master server at 127.21.98.254:36033
I20250905 14:00:06.131981 28473 heartbeater.cc:461] Registering TS with master...
I20250905 14:00:06.133121 28473 heartbeater.cc:507] Master 127.21.98.254:36033 requested a full tablet report, sending...
I20250905 14:00:06.137717 28286 ts_manager.cc:194] Registered new tserver with Master: e4bef33a2dbd44768749b509e0d53995 (127.21.98.193:37031)
I20250905 14:00:06.145190 28286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.193:32867
I20250905 14:00:06.174425 28366 log.cc:826] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Log is configured to *not* fsync() on all Append() calls
W20250905 14:00:06.531107 28474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 14:00:06.531752 28474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 14:00:06.532341 28474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 14:00:06.562399 28474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 14:00:06.563247 28474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.194
I20250905 14:00:06.596937 28474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.194:41267
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.21.98.194
--webserver_port=43529
--tserver_master_addrs=127.21.98.254:36033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.194
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 14:00:06.598201 28474 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 14:00:06.599825 28474 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 14:00:06.612694 28484 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 14:00:07.150489 28473 heartbeater.cc:499] Master 127.21.98.254:36033 was elected leader, sending a full tablet report...
W20250905 14:00:08.015872 28483 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 28474
W20250905 14:00:08.371438 28474 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.758s user 0.649s sys 1.015s
W20250905 14:00:08.372236 28486 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1757 milliseconds
W20250905 14:00:06.615432 28485 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 14:00:08.372506 28474 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.759s user 0.649s sys 1.015s
I20250905 14:00:08.373528 28474 server_base.cc:1042] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
W20250905 14:00:08.373589 28487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 14:00:08.377308 28474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 14:00:08.379499 28474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 14:00:08.380873 28474 hybrid_clock.cc:648] HybridClock initialized: now 1757080808380821 us; error 49 us; skew 500 ppm
I20250905 14:00:08.381629 28474 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 14:00:08.388290 28474 webserver.cc:480] Webserver started at http://127.21.98.194:43529/ using document root <none> and password file <none>
I20250905 14:00:08.389150 28474 fs_manager.cc:362] Metadata directory not provided
I20250905 14:00:08.389381 28474 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 14:00:08.397336 28474 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.001s sys 0.005s
I20250905 14:00:08.402052 28495 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 14:00:08.403098 28474 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20250905 14:00:08.403406 28474 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal
uuid: "f62993cab8bf4bc7b34b11fae92969a2"
format_stamp: "Formatted at 2025-09-05 13:59:50 on dist-test-slave-g001"
I20250905 14:00:08.405267 28474 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 14:00:08.457024 28474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 14:00:08.458402 28474 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 14:00:08.458843 28474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 14:00:08.461620 28474 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 14:00:08.467581 28502 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 14:00:08.476248 28474 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 14:00:08.476454 28474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.011s user 0.002s sys 0.000s
I20250905 14:00:08.476722 28474 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 14:00:08.481035 28474 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 14:00:08.481274 28474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.003s sys 0.000s
I20250905 14:00:08.481631 28502 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Bootstrap starting.
I20250905 14:00:08.659726 28474 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.194:41267
I20250905 14:00:08.659868 28608 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.194:41267 every 8 connection(s)
I20250905 14:00:08.663480 28474 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/info.pb
I20250905 14:00:08.669929 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 28474
I20250905 14:00:08.672079 21899 external_mini_cluster.cc:1366] Running /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
/tmp/dist-test-task46B7ZV/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.21.98.195:44685
--local_ip_for_outbound_sockets=127.21.98.195
--tserver_master_addrs=127.21.98.254:36033
--webserver_port=45537
--webserver_interface=127.21.98.195
--builtin_ntp_servers=127.21.98.212:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20250905 14:00:08.712834 28609 heartbeater.cc:344] Connected to a master server at 127.21.98.254:36033
I20250905 14:00:08.713348 28609 heartbeater.cc:461] Registering TS with master...
I20250905 14:00:08.714620 28609 heartbeater.cc:507] Master 127.21.98.254:36033 requested a full tablet report, sending...
I20250905 14:00:08.718467 28286 ts_manager.cc:194] Registered new tserver with Master: f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194:41267)
I20250905 14:00:08.721460 28286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.194:40163
I20250905 14:00:08.797930 28502 log.cc:826] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Log is configured to *not* fsync() on all Append() calls
W20250905 14:00:09.140141 28613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 14:00:09.140724 28613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 14:00:09.141392 28613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 14:00:09.194164 28613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 14:00:09.195567 28613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.21.98.195
I20250905 14:00:09.253269 28613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.21.98.212:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.21.98.195:44685
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.21.98.195
--webserver_port=45537
--tserver_master_addrs=127.21.98.254:36033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.21.98.195
--log_dir=/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 13:43:21 UTC on 5fd53c4cbb9d
build id 7913
TSAN enabled
I20250905 14:00:09.254860 28613 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 14:00:09.257067 28613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 14:00:09.274384 28620 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 14:00:09.589586 28366 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 14:00:09.591042 28366 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Bootstrap complete.
I20250905 14:00:09.593493 28366 ts_tablet_manager.cc:1397] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Time spent bootstrapping tablet: real 3.722s user 3.583s sys 0.095s
I20250905 14:00:09.615069 28366 raft_consensus.cc:357] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:09.619240 28366 raft_consensus.cc:738] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bef33a2dbd44768749b509e0d53995, State: Initialized, Role: FOLLOWER
I20250905 14:00:09.620524 28366 consensus_queue.cc:260] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:09.630826 28366 ts_tablet_manager.cc:1428] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Time spent starting tablet: real 0.037s user 0.027s sys 0.006s
I20250905 14:00:09.725241 28609 heartbeater.cc:499] Master 127.21.98.254:36033 was elected leader, sending a full tablet report...
I20250905 14:00:10.843144 28502 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 14:00:10.843907 28502 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Bootstrap complete.
I20250905 14:00:10.845286 28502 ts_tablet_manager.cc:1397] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Time spent bootstrapping tablet: real 2.364s user 2.268s sys 0.076s
I20250905 14:00:10.855083 28502 raft_consensus.cc:357] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:10.857133 28502 raft_consensus.cc:738] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62993cab8bf4bc7b34b11fae92969a2, State: Initialized, Role: FOLLOWER
I20250905 14:00:10.857856 28502 consensus_queue.cc:260] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:10.861342 28502 ts_tablet_manager.cc:1428] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
W20250905 14:00:10.673393 28619 debug-util.cc:398] Leaking SignalData structure 0x7b0800034ea0 after lost signal to thread 28613
W20250905 14:00:09.274580 28621 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 14:00:10.875993 28613 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.603s user 0.604s sys 0.986s
W20250905 14:00:10.876365 28613 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.604s user 0.605s sys 0.987s
W20250905 14:00:10.883543 28627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 14:00:10.884464 28613 server_base.cc:1047] running on GCE node
I20250905 14:00:10.885461 28613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 14:00:10.887389 28613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 14:00:10.888703 28613 hybrid_clock.cc:648] HybridClock initialized: now 1757080810888662 us; error 40 us; skew 500 ppm
I20250905 14:00:10.889407 28613 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 14:00:10.894595 28613 webserver.cc:480] Webserver started at http://127.21.98.195:45537/ using document root <none> and password file <none>
I20250905 14:00:10.895502 28613 fs_manager.cc:362] Metadata directory not provided
I20250905 14:00:10.895718 28613 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 14:00:10.902912 28613 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20250905 14:00:10.907295 28632 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 14:00:10.908164 28613 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250905 14:00:10.908438 28613 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data,/tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal
uuid: "62fc44fafab2486e99bd94370b83619e"
format_stamp: "Formatted at 2025-09-05 13:59:52 on dist-test-slave-g001"
I20250905 14:00:10.910199 28613 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 14:00:10.953519 28613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 14:00:10.954854 28613 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 14:00:10.955296 28613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 14:00:10.958357 28613 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 14:00:10.965106 28639 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20250905 14:00:10.972975 28613 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20250905 14:00:10.973323 28613 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.010s user 0.002s sys 0.000s
I20250905 14:00:10.973598 28613 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20250905 14:00:10.977988 28613 ts_tablet_manager.cc:610] Registered 1 tablets
I20250905 14:00:10.978241 28613 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.002s sys 0.000s
I20250905 14:00:10.978593 28639 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Bootstrap starting.
I20250905 14:00:11.130489 28613 rpc_server.cc:307] RPC server started. Bound to: 127.21.98.195:44685
I20250905 14:00:11.130726 28745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.21.98.195:44685 every 8 connection(s)
I20250905 14:00:11.133126 28613 server_base.cc:1179] Dumped server information to /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/info.pb
I20250905 14:00:11.142407 21899 external_mini_cluster.cc:1428] Started /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu as pid 28613
I20250905 14:00:11.177757 28746 heartbeater.cc:344] Connected to a master server at 127.21.98.254:36033
I20250905 14:00:11.178224 28746 heartbeater.cc:461] Registering TS with master...
I20250905 14:00:11.179459 28746 heartbeater.cc:507] Master 127.21.98.254:36033 requested a full tablet report, sending...
I20250905 14:00:11.182523 28286 ts_manager.cc:194] Registered new tserver with Master: 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685)
I20250905 14:00:11.184720 28286 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.21.98.195:42127
I20250905 14:00:11.194214 21899 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 14:00:11.253768 28639 log.cc:826] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Log is configured to *not* fsync() on all Append() calls
I20250905 14:00:11.363556 28758 raft_consensus.cc:491] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 14:00:11.364018 28758 raft_consensus.cc:513] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:11.366402 28758 leader_election.cc:290] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194:41267), 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685)
I20250905 14:00:11.396559 28564 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a6e84b9ab724881a41dc83db6bafe95" candidate_uuid: "e4bef33a2dbd44768749b509e0d53995" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f62993cab8bf4bc7b34b11fae92969a2" is_pre_election: true
I20250905 14:00:11.397255 28564 raft_consensus.cc:2466] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4bef33a2dbd44768749b509e0d53995 in term 1.
I20250905 14:00:11.398788 28362 leader_election.cc:304] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4bef33a2dbd44768749b509e0d53995, f62993cab8bf4bc7b34b11fae92969a2; no voters:
I20250905 14:00:11.401715 28758 raft_consensus.cc:2802] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 14:00:11.402014 28758 raft_consensus.cc:491] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 14:00:11.402273 28758 raft_consensus.cc:3058] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Advancing to term 2
I20250905 14:00:11.397094 28701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a6e84b9ab724881a41dc83db6bafe95" candidate_uuid: "e4bef33a2dbd44768749b509e0d53995" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "62fc44fafab2486e99bd94370b83619e" is_pre_election: true
W20250905 14:00:11.408797 28360 leader_election.cc:343] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685): Illegal state: must be running to vote when last-logged opid is not known
I20250905 14:00:11.411434 28758 raft_consensus.cc:513] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:11.413553 28758 leader_election.cc:290] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [CANDIDATE]: Term 2 election: Requested vote from peers f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194:41267), 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685)
I20250905 14:00:11.413815 28564 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a6e84b9ab724881a41dc83db6bafe95" candidate_uuid: "e4bef33a2dbd44768749b509e0d53995" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "f62993cab8bf4bc7b34b11fae92969a2"
I20250905 14:00:11.414098 28701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a6e84b9ab724881a41dc83db6bafe95" candidate_uuid: "e4bef33a2dbd44768749b509e0d53995" candidate_term: 2 candidate_status { last_received { term: 1 index: 205 } } ignore_live_leader: false dest_uuid: "62fc44fafab2486e99bd94370b83619e"
I20250905 14:00:11.414373 28564 raft_consensus.cc:3058] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 1 FOLLOWER]: Advancing to term 2
W20250905 14:00:11.415218 28360 leader_election.cc:343] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685): Illegal state: must be running to vote when last-logged opid is not known
I20250905 14:00:11.420498 28564 raft_consensus.cc:2466] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bef33a2dbd44768749b509e0d53995 in term 2.
I20250905 14:00:11.421221 28362 leader_election.cc:304] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e4bef33a2dbd44768749b509e0d53995, f62993cab8bf4bc7b34b11fae92969a2; no voters: 62fc44fafab2486e99bd94370b83619e
I20250905 14:00:11.421800 28758 raft_consensus.cc:2802] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 14:00:11.423341 28758 raft_consensus.cc:695] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 2 LEADER]: Becoming Leader. State: Replica: e4bef33a2dbd44768749b509e0d53995, State: Running, Role: LEADER
I20250905 14:00:11.424268 28758 consensus_queue.cc:237] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 205, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:11.436187 28286 catalog_manager.cc:5582] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 reported cstate change: term changed from 0 to 2, leader changed from <none> to e4bef33a2dbd44768749b509e0d53995 (127.21.98.193), VOTER 62fc44fafab2486e99bd94370b83619e (127.21.98.195) added, VOTER e4bef33a2dbd44768749b509e0d53995 (127.21.98.193) added, VOTER f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194) added. New cstate: current_term: 2 leader_uuid: "e4bef33a2dbd44768749b509e0d53995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } health_report { overall_health: UNKNOWN } } }
W20250905 14:00:11.877429 21899 scanner-internal.cc:458] Time spent opening tablet: real 0.655s user 0.006s sys 0.000s
I20250905 14:00:11.980798 28564 raft_consensus.cc:1273] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 2 FOLLOWER]: Refusing update from remote peer e4bef33a2dbd44768749b509e0d53995: Log matching property violated. Preceding OpId in replica: term: 1 index: 205. Preceding OpId from leader: term: 2 index: 206. (index mismatch)
I20250905 14:00:11.982968 28772 consensus_queue.cc:1035] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 206, Last known committed idx: 205, Time since last communication: 0.001s
W20250905 14:00:12.009707 28360 consensus_peers.cc:489] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 -> Peer 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685): Couldn't send request to peer 62fc44fafab2486e99bd94370b83619e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250905 14:00:12.104905 28428 consensus_queue.cc:237] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 206, Committed index: 206, Last appended: 2.206, Last appended by leader: 205, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } }
I20250905 14:00:12.110126 28564 raft_consensus.cc:1273] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 2 FOLLOWER]: Refusing update from remote peer e4bef33a2dbd44768749b509e0d53995: Log matching property violated. Preceding OpId in replica: term: 2 index: 206. Preceding OpId from leader: term: 2 index: 207. (index mismatch)
I20250905 14:00:12.111613 28778 consensus_queue.cc:1035] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 207, Last known committed idx: 206, Time since last communication: 0.000s
I20250905 14:00:12.118068 28778 raft_consensus.cc:2953] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 2 LEADER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 62fc44fafab2486e99bd94370b83619e (127.21.98.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } }
I20250905 14:00:12.120461 28564 raft_consensus.cc:2953] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 2 FOLLOWER]: Committing config change with OpId 2.207: config changed from index -1 to 207, VOTER 62fc44fafab2486e99bd94370b83619e (127.21.98.195) evicted. New config: { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } }
I20250905 14:00:12.136561 28272 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5a6e84b9ab724881a41dc83db6bafe95 with cas_config_opid_index -1: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 14:00:12.142298 28286 catalog_manager.cc:5582] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 reported cstate change: config changed from index -1 to 207, VOTER 62fc44fafab2486e99bd94370b83619e (127.21.98.195) evicted. New cstate: current_term: 2 leader_uuid: "e4bef33a2dbd44768749b509e0d53995" committed_config { opid_index: 207 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } health_report { overall_health: HEALTHY } } }
I20250905 14:00:12.187513 28746 heartbeater.cc:499] Master 127.21.98.254:36033 was elected leader, sending a full tablet report...
I20250905 14:00:12.203594 28428 consensus_queue.cc:237] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 207, Committed index: 207, Last appended: 2.207, Last appended by leader: 205, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } }
I20250905 14:00:12.206487 28778 raft_consensus.cc:2953] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 2 LEADER]: Committing config change with OpId 2.208: config changed from index 207 to 208, VOTER f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194) evicted. New config: { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } }
I20250905 14:00:12.219413 28272 catalog_manager.cc:5095] ChangeConfig:REMOVE_PEER RPC for tablet 5a6e84b9ab724881a41dc83db6bafe95 with cas_config_opid_index 207: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20250905 14:00:12.223429 28285 catalog_manager.cc:5582] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 reported cstate change: config changed from index 207 to 208, VOTER f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194) evicted. New cstate: current_term: 2 leader_uuid: "e4bef33a2dbd44768749b509e0d53995" committed_config { opid_index: 208 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } health_report { overall_health: HEALTHY } } }
I20250905 14:00:12.252455 28681 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a6e84b9ab724881a41dc83db6bafe95 with delete_type TABLET_DATA_TOMBSTONED (TS 62fc44fafab2486e99bd94370b83619e not found in new config with opid_index 207) from {username='slave'} at 127.0.0.1:59644
W20250905 14:00:12.261054 28270 catalog_manager.cc:4908] TS 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685): delete failed for tablet 5a6e84b9ab724881a41dc83db6bafe95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5a6e84b9ab724881a41dc83db6bafe95 already in progress: opening tablet
I20250905 14:00:12.264045 28544 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a6e84b9ab724881a41dc83db6bafe95 with delete_type TABLET_DATA_TOMBSTONED (TS f62993cab8bf4bc7b34b11fae92969a2 not found in new config with opid_index 208) from {username='slave'} at 127.0.0.1:51558
I20250905 14:00:12.276335 28788 tablet_replica.cc:331] stopping tablet replica
I20250905 14:00:12.277190 28788 raft_consensus.cc:2241] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 14:00:12.277854 28788 raft_consensus.cc:2270] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 14:00:12.304220 28788 ts_tablet_manager.cc:1905] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 14:00:12.320035 28788 ts_tablet_manager.cc:1918] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.207
I20250905 14:00:12.320760 28788 log.cc:1199] T 5a6e84b9ab724881a41dc83db6bafe95 P f62993cab8bf4bc7b34b11fae92969a2: Deleting WAL directory at /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/wal/wals/5a6e84b9ab724881a41dc83db6bafe95
I20250905 14:00:12.323063 28272 catalog_manager.cc:4928] TS f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194:41267): tablet 5a6e84b9ab724881a41dc83db6bafe95 (table pre_rebuild [id=1020fee07c1643b1bcda03bd22e574e6]) successfully deleted
I20250905 14:00:12.829936 28681 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 14:00:12.860810 28408 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 14:00:12.892282 28544 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
a53cd829c39a47bfaba2917be9d0645c | 127.21.98.254:36033 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.98.212:32833 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
62fc44fafab2486e99bd94370b83619e | 127.21.98.195:44685 | HEALTHY | <none> | 0 | 0
e4bef33a2dbd44768749b509e0d53995 | 127.21.98.193:37031 | HEALTHY | <none> | 1 | 0
f62993cab8bf4bc7b34b11fae92969a2 | 127.21.98.194:41267 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.21.98.193 | experimental | 127.21.98.193:37031
local_ip_for_outbound_sockets | 127.21.98.194 | experimental | 127.21.98.194:41267
local_ip_for_outbound_sockets | 127.21.98.195 | experimental | 127.21.98.195:44685
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.98.193:37031
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.98.194:41267
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/info.pb | hidden | 127.21.98.195:44685
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.98.212:32833 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------+----+---------+---------------+---------+------------+------------------+-------------
pre_rebuild | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 0
First Quartile | 0
Median | 0
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 1
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250905 14:00:13.229815 21899 log_verifier.cc:126] Checking tablet 5a6e84b9ab724881a41dc83db6bafe95
I20250905 14:00:13.572619 28639 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Bootstrap replayed 1/1 log segments. Stats: ops{read=205 overwritten=0 applied=205 ignored=0} inserts{seen=10200 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 14:00:13.573406 28639 tablet_bootstrap.cc:492] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Bootstrap complete.
I20250905 14:00:13.574661 28639 ts_tablet_manager.cc:1397] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Time spent bootstrapping tablet: real 2.596s user 2.407s sys 0.071s
I20250905 14:00:13.579704 28639 raft_consensus.cc:357] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:13.581382 28639 raft_consensus.cc:738] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fc44fafab2486e99bd94370b83619e, State: Initialized, Role: FOLLOWER
I20250905 14:00:13.582021 28639 consensus_queue.cc:260] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 205, Last appended: 1.205, Last appended by leader: 205, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:13.585907 28639 ts_tablet_manager.cc:1428] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20250905 14:00:13.588969 28681 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a6e84b9ab724881a41dc83db6bafe95 with delete_type TABLET_DATA_TOMBSTONED (Replica has no consensus available (current committed config index is 208)) from {username='slave'} at 127.0.0.1:59644
I20250905 14:00:13.595669 28816 tablet_replica.cc:331] stopping tablet replica
I20250905 14:00:13.596415 28816 raft_consensus.cc:2241] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 14:00:13.596923 28816 raft_consensus.cc:2270] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 14:00:13.611073 28816 ts_tablet_manager.cc:1905] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20250905 14:00:13.623787 28816 ts_tablet_manager.cc:1918] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.205
I20250905 14:00:13.624143 28816 log.cc:1199] T 5a6e84b9ab724881a41dc83db6bafe95 P 62fc44fafab2486e99bd94370b83619e: Deleting WAL directory at /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/wal/wals/5a6e84b9ab724881a41dc83db6bafe95
I20250905 14:00:13.625644 28270 catalog_manager.cc:4928] TS 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685): tablet 5a6e84b9ab724881a41dc83db6bafe95 (table pre_rebuild [id=1020fee07c1643b1bcda03bd22e574e6]) successfully deleted
I20250905 14:00:13.741988 21899 log_verifier.cc:177] Verified matching terms for 208 ops in tablet 5a6e84b9ab724881a41dc83db6bafe95
I20250905 14:00:13.744800 28286 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56010:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250905 14:00:13.745546 28286 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56010:
table { table_name: "pre_rebuild" } modify_external_catalogs: true
I20250905 14:00:13.760102 28286 catalog_manager.cc:5869] T 00000000000000000000000000000000 P a53cd829c39a47bfaba2917be9d0645c: Sending DeleteTablet for 1 replicas of tablet 5a6e84b9ab724881a41dc83db6bafe95
I20250905 14:00:13.761845 21899 test_util.cc:276] Using random seed: 1103033629
I20250905 14:00:13.761787 28408 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a6e84b9ab724881a41dc83db6bafe95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 14:00:13 UTC) from {username='slave'} at 127.0.0.1:43122
I20250905 14:00:13.763536 28821 tablet_replica.cc:331] stopping tablet replica
I20250905 14:00:13.764129 28821 raft_consensus.cc:2241] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 2 LEADER]: Raft consensus shutting down.
I20250905 14:00:13.764604 28821 raft_consensus.cc:2270] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 14:00:13.796801 28821 ts_tablet_manager.cc:1905] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 14:00:13.799149 28286 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56042:
name: "post_rebuild"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 14:00:13.802606 28286 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table post_rebuild in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 14:00:13.807996 28821 ts_tablet_manager.cc:1918] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.208
I20250905 14:00:13.808393 28821 log.cc:1199] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Deleting WAL directory at /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/wal/wals/5a6e84b9ab724881a41dc83db6bafe95
I20250905 14:00:13.809203 28821 ts_tablet_manager.cc:1939] T 5a6e84b9ab724881a41dc83db6bafe95 P e4bef33a2dbd44768749b509e0d53995: Deleting consensus metadata
I20250905 14:00:13.811843 28272 catalog_manager.cc:4928] TS e4bef33a2dbd44768749b509e0d53995 (127.21.98.193:37031): tablet 5a6e84b9ab724881a41dc83db6bafe95 (table pre_rebuild [id=1020fee07c1643b1bcda03bd22e574e6]) successfully deleted
I20250905 14:00:13.827432 28544 tablet_service.cc:1468] Processing CreateTablet for tablet 7fea05babcb64bd18e48573f33346422 (DEFAULT_TABLE table=post_rebuild [id=0ab5a2a196f04ccfaf1d31b5ec4675b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 14:00:13.827746 28408 tablet_service.cc:1468] Processing CreateTablet for tablet 7fea05babcb64bd18e48573f33346422 (DEFAULT_TABLE table=post_rebuild [id=0ab5a2a196f04ccfaf1d31b5ec4675b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 14:00:13.828831 28408 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fea05babcb64bd18e48573f33346422. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 14:00:13.828804 28544 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fea05babcb64bd18e48573f33346422. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 14:00:13.833295 28681 tablet_service.cc:1468] Processing CreateTablet for tablet 7fea05babcb64bd18e48573f33346422 (DEFAULT_TABLE table=post_rebuild [id=0ab5a2a196f04ccfaf1d31b5ec4675b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 14:00:13.834611 28681 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fea05babcb64bd18e48573f33346422. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 14:00:13.848953 28639 tablet_bootstrap.cc:492] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e: Bootstrap starting.
I20250905 14:00:13.850646 28828 tablet_bootstrap.cc:492] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995: Bootstrap starting.
I20250905 14:00:13.853992 28639 tablet_bootstrap.cc:654] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e: Neither blocks nor log segments found. Creating new log.
I20250905 14:00:13.856375 28828 tablet_bootstrap.cc:654] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995: Neither blocks nor log segments found. Creating new log.
I20250905 14:00:13.860028 28829 tablet_bootstrap.cc:492] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2: Bootstrap starting.
I20250905 14:00:13.867518 28829 tablet_bootstrap.cc:654] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2: Neither blocks nor log segments found. Creating new log.
I20250905 14:00:13.868312 28828 tablet_bootstrap.cc:492] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995: No bootstrap required, opened a new log
I20250905 14:00:13.868769 28828 ts_tablet_manager.cc:1397] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.008s
I20250905 14:00:13.871505 28828 raft_consensus.cc:357] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:13.872354 28639 tablet_bootstrap.cc:492] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e: No bootstrap required, opened a new log
I20250905 14:00:13.872287 28828 raft_consensus.cc:383] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 14:00:13.872565 28828 raft_consensus.cc:738] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4bef33a2dbd44768749b509e0d53995, State: Initialized, Role: FOLLOWER
I20250905 14:00:13.872767 28639 ts_tablet_manager.cc:1397] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.008s
I20250905 14:00:13.873311 28828 consensus_queue.cc:260] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:13.878872 28639 raft_consensus.cc:357] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:13.879669 28639 raft_consensus.cc:383] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 14:00:13.879967 28639 raft_consensus.cc:738] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62fc44fafab2486e99bd94370b83619e, State: Initialized, Role: FOLLOWER
I20250905 14:00:13.880625 28639 consensus_queue.cc:260] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:13.882356 28639 ts_tablet_manager.cc:1428] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250905 14:00:13.884886 28828 ts_tablet_manager.cc:1428] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995: Time spent starting tablet: real 0.016s user 0.005s sys 0.007s
I20250905 14:00:13.885044 28829 tablet_bootstrap.cc:492] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2: No bootstrap required, opened a new log
I20250905 14:00:13.885474 28829 ts_tablet_manager.cc:1397] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2: Time spent bootstrapping tablet: real 0.026s user 0.017s sys 0.007s
I20250905 14:00:13.888859 28829 raft_consensus.cc:357] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:13.889734 28829 raft_consensus.cc:383] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 14:00:13.890041 28829 raft_consensus.cc:738] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f62993cab8bf4bc7b34b11fae92969a2, State: Initialized, Role: FOLLOWER
W20250905 14:00:13.890973 28747 tablet.cc:2378] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 14:00:13.890771 28829 consensus_queue.cc:260] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:13.893630 28829 ts_tablet_manager.cc:1428] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
W20250905 14:00:13.938407 28610 tablet.cc:2378] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 14:00:13.996799 28833 raft_consensus.cc:491] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 14:00:13.997190 28833 raft_consensus.cc:513] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:13.998832 28833 leader_election.cc:290] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194:41267), 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685)
I20250905 14:00:13.999595 28564 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fea05babcb64bd18e48573f33346422" candidate_uuid: "e4bef33a2dbd44768749b509e0d53995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62993cab8bf4bc7b34b11fae92969a2" is_pre_election: true
I20250905 14:00:14.000090 28564 raft_consensus.cc:2466] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4bef33a2dbd44768749b509e0d53995 in term 0.
I20250905 14:00:14.000008 28701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fea05babcb64bd18e48573f33346422" candidate_uuid: "e4bef33a2dbd44768749b509e0d53995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fc44fafab2486e99bd94370b83619e" is_pre_election: true
I20250905 14:00:14.000671 28701 raft_consensus.cc:2466] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e4bef33a2dbd44768749b509e0d53995 in term 0.
I20250905 14:00:14.000900 28362 leader_election.cc:304] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4bef33a2dbd44768749b509e0d53995, f62993cab8bf4bc7b34b11fae92969a2; no voters:
I20250905 14:00:14.001446 28833 raft_consensus.cc:2802] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 14:00:14.001711 28833 raft_consensus.cc:491] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 14:00:14.001960 28833 raft_consensus.cc:3058] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 0 FOLLOWER]: Advancing to term 1
I20250905 14:00:14.008173 28833 raft_consensus.cc:513] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:14.009864 28833 leader_election.cc:290] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [CANDIDATE]: Term 1 election: Requested vote from peers f62993cab8bf4bc7b34b11fae92969a2 (127.21.98.194:41267), 62fc44fafab2486e99bd94370b83619e (127.21.98.195:44685)
I20250905 14:00:14.010149 28564 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fea05babcb64bd18e48573f33346422" candidate_uuid: "e4bef33a2dbd44768749b509e0d53995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f62993cab8bf4bc7b34b11fae92969a2"
I20250905 14:00:14.010511 28564 raft_consensus.cc:3058] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2 [term 0 FOLLOWER]: Advancing to term 1
I20250905 14:00:14.010408 28701 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7fea05babcb64bd18e48573f33346422" candidate_uuid: "e4bef33a2dbd44768749b509e0d53995" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "62fc44fafab2486e99bd94370b83619e"
I20250905 14:00:14.010838 28701 raft_consensus.cc:3058] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e [term 0 FOLLOWER]: Advancing to term 1
I20250905 14:00:14.014361 28564 raft_consensus.cc:2466] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bef33a2dbd44768749b509e0d53995 in term 1.
I20250905 14:00:14.015105 28362 leader_election.cc:304] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e4bef33a2dbd44768749b509e0d53995, f62993cab8bf4bc7b34b11fae92969a2; no voters:
I20250905 14:00:14.015632 28833 raft_consensus.cc:2802] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 14:00:14.017122 28701 raft_consensus.cc:2466] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e4bef33a2dbd44768749b509e0d53995 in term 1.
I20250905 14:00:14.016781 28833 raft_consensus.cc:695] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [term 1 LEADER]: Becoming Leader. State: Replica: e4bef33a2dbd44768749b509e0d53995, State: Running, Role: LEADER
I20250905 14:00:14.017974 28833 consensus_queue.cc:237] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } } peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } }
I20250905 14:00:14.023495 28285 catalog_manager.cc:5582] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 reported cstate change: term changed from 0 to 1, leader changed from <none> to e4bef33a2dbd44768749b509e0d53995 (127.21.98.193). New cstate: current_term: 1 leader_uuid: "e4bef33a2dbd44768749b509e0d53995" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e4bef33a2dbd44768749b509e0d53995" member_type: VOTER last_known_addr { host: "127.21.98.193" port: 37031 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 } health_report { overall_health: UNKNOWN } } }
W20250905 14:00:14.120862 28475 tablet.cc:2378] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 14:00:14.210628 28701 raft_consensus.cc:1273] T 7fea05babcb64bd18e48573f33346422 P 62fc44fafab2486e99bd94370b83619e [term 1 FOLLOWER]: Refusing update from remote peer e4bef33a2dbd44768749b509e0d53995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 14:00:14.212513 28833 consensus_queue.cc:1035] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62fc44fafab2486e99bd94370b83619e" member_type: VOTER last_known_addr { host: "127.21.98.195" port: 44685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 14:00:14.221868 28564 raft_consensus.cc:1273] T 7fea05babcb64bd18e48573f33346422 P f62993cab8bf4bc7b34b11fae92969a2 [term 1 FOLLOWER]: Refusing update from remote peer e4bef33a2dbd44768749b509e0d53995: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 14:00:14.223572 28833 consensus_queue.cc:1035] T 7fea05babcb64bd18e48573f33346422 P e4bef33a2dbd44768749b509e0d53995 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f62993cab8bf4bc7b34b11fae92969a2" member_type: VOTER last_known_addr { host: "127.21.98.194" port: 41267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 14:00:14.267251 28843 mvcc.cc:204] Tried to move back new op lower bound from 7197003014986260480 to 7197003014223253504. Current Snapshot: MvccSnapshot[applied={T|T < 7197003014986260480}]
I20250905 14:00:14.281980 28845 mvcc.cc:204] Tried to move back new op lower bound from 7197003014986260480 to 7197003014223253504. Current Snapshot: MvccSnapshot[applied={T|T < 7197003014986260480 or (T in {7197003014986260480})}]
I20250905 14:00:18.723399 28681 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
I20250905 14:00:18.734833 28408 tablet_service.cc:1430] Tablet server has 1 leaders and 0 scanners
I20250905 14:00:18.735275 28544 tablet_service.cc:1430] Tablet server has 0 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+---------------------+---------
a53cd829c39a47bfaba2917be9d0645c | 127.21.98.254:36033 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.98.212:32833 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+---------------------+---------+----------+----------------+-----------------
62fc44fafab2486e99bd94370b83619e | 127.21.98.195:44685 | HEALTHY | <none> | 0 | 0
e4bef33a2dbd44768749b509e0d53995 | 127.21.98.193:37031 | HEALTHY | <none> | 1 | 0
f62993cab8bf4bc7b34b11fae92969a2 | 127.21.98.194:41267 | HEALTHY | <none> | 0 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 3
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------+-------------------------
ipki_server_key_size | 768 | experimental | all 3 server(s) checked
local_ip_for_outbound_sockets | 127.21.98.193 | experimental | 127.21.98.193:37031
local_ip_for_outbound_sockets | 127.21.98.194 | experimental | 127.21.98.194:41267
local_ip_for_outbound_sockets | 127.21.98.195 | experimental | 127.21.98.195:44685
never_fsync | true | unsafe,advanced | all 3 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 3 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 3 server(s) checked
server_dump_info_format | pb | hidden | all 3 server(s) checked
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-0/data/info.pb | hidden | 127.21.98.193:37031
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-1/data/info.pb | hidden | 127.21.98.194:41267
server_dump_info_path | /tmp/dist-test-task46B7ZV/test-tmp/kudu-admin-test.5.IsSecure_SecureClusterAdminCliParamTest.TestRebuildMaster_0.1757080637411871-21899-0/minicluster-data/ts-2/data/info.pb | hidden | 127.21.98.195:44685
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+---------------------+-------------------------
builtin_ntp_servers | 127.21.98.212:32833 | all 3 server(s) checked
time_source | builtin | all 3 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.19.0-SNAPSHOT | all 4 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
--------------+----+---------+---------------+---------+------------+------------------+-------------
post_rebuild | 3 | HEALTHY | 1 | 1 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 1
First Quartile | 1
Median | 1
Third Quartile | 1
Maximum | 1
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 3
Tables | 1
Tablets | 1
Replicas | 3
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20250905 14:00:18.934996 21899 log_verifier.cc:126] Checking tablet 5a6e84b9ab724881a41dc83db6bafe95
I20250905 14:00:18.935539 21899 log_verifier.cc:177] Verified matching terms for 0 ops in tablet 5a6e84b9ab724881a41dc83db6bafe95
I20250905 14:00:18.935799 21899 log_verifier.cc:126] Checking tablet 7fea05babcb64bd18e48573f33346422
I20250905 14:00:19.645602 21899 log_verifier.cc:177] Verified matching terms for 205 ops in tablet 7fea05babcb64bd18e48573f33346422
I20250905 14:00:19.676699 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 28323
I20250905 14:00:19.719056 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 28474
I20250905 14:00:19.755066 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 28613
I20250905 14:00:19.788852 21899 external_mini_cluster.cc:1658] Killing /tmp/dist-test-task46B7ZV/build/tsan/bin/kudu with pid 28253
2025-09-05T14:00:19Z chronyd exiting
[ OK ] IsSecure/SecureClusterAdminCliParamTest.TestRebuildMaster/0 (35513 ms)
[----------] 1 test from IsSecure/SecureClusterAdminCliParamTest (35513 ms total)
[----------] Global test environment tear-down
[==========] 9 tests from 5 test suites ran. (182367 ms total)
[ PASSED ] 8 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AdminCliTest.TestRebuildTables
1 FAILED TEST